bqext

README

The bqext package

This package contains extensions for the bigquery library, to facilitate various common operations, notably query processing.

Documentation

Overview

Package bqext includes generally useful abstractions for simplifying interactions with bigquery. Production extensions should go here, but test facilities should go in a separate bqtest package.

Index

• type Dataset
  • func NewDataset(project, dataset string, clientOpts ...option.ClientOption) (Dataset, error)
  • func (dsExt *Dataset) Dedup(src string, dedupOn string, overwrite bool, project, dataset, table string) (*bigquery.JobStatus, error)
  • func (dsExt *Dataset) DestinationQuery(query string, dest *bigquery.Table) *bigquery.Query
  • func (dsExt Dataset) GetPartitionInfo(table string, partition string) (PartitionInfo, error)
  • func (dsExt *Dataset) QueryAndParse(q string, structPtr interface{}) error
  • func (dsExt *Dataset) ResultQuery(query string, dryRun bool) *bigquery.Query
• type PartitionInfo

Types

type Dataset

type Dataset struct {
	BqClient *bigquery.Client
	Dataset  *bigquery.Dataset
}

Dataset provides extensions to the bigquery Dataset and Dataset objects to streamline common actions. It encapsulates the Client and Dataset to simplify methods.

func NewDataset

func NewDataset(project, dataset string, clientOpts ...option.ClientOption) (Dataset, error)

NewDataset creates a Dataset for a project. httpClient is used to inject mocks for the bigquery client. if httpClient is nil, a suitable default client is used. Additional bigquery ClientOptions may be optionally passed as final

clientOpts argument.  This is useful for testing credentials.

func (*Dataset) Dedup

func (dsExt *Dataset) Dedup(src string, dedupOn string, overwrite bool, project, dataset, table string) (*bigquery.JobStatus, error)

Dedup executes a query that dedups and writes to an appropriate partition. src is relative to the project:dataset of dsExt. dedupOn names the field to be used for dedupping. project, dataset, table specify the table to write into. NOTE: destination table must include the partition suffix. This avoids accidentally overwriting TODAY's partition.

func (*Dataset) DestinationQuery

func (dsExt *Dataset) DestinationQuery(query string, dest *bigquery.Table) *bigquery.Query

DestinationQuery constructs a query with common Config settings for writing results to a table. Generally, may need to change WriteDisposition.

func (Dataset) GetPartitionInfo

func (dsExt Dataset) GetPartitionInfo(table string, partition string) (PartitionInfo, error)

GetPartitionInfo provides basic information about a partition.

func (*Dataset) QueryAndParse

func (dsExt *Dataset) QueryAndParse(q string, structPtr interface{}) error

QueryAndParse executes a query that should return a single row, with all struct fields that match query columns filled in. The caller must pass in the *address* of an appropriate struct. TODO - extend this to also handle multirow results, by passing slice of structs.

func (*Dataset) ResultQuery

func (dsExt *Dataset) ResultQuery(query string, dryRun bool) *bigquery.Query

ResultQuery constructs a query with common QueryConfig settings for writing results to a table. Generally, may need to change WriteDisposition.

type PartitionInfo

type PartitionInfo struct {
	PartitionID  string
	CreationTime time.Time
	LastModified time.Time
}

PartitionInfo provides basic information about a partition.