bqext

README

The bqext package

This package contains extensions for the bigquery library, to facilitate various common operations, notably query processing.

Documentation

Overview

Package bqext includes generally useful abstractions for simplifying interactions with bigquery. Production extensions should go here, but test facilities should go in a separate bqtest package.

Index

• type Dataset
  • func NewDataset(project, dataset string, clientOpts ...option.ClientOption) (Dataset, error)
  • func (dsExt *Dataset) Dedup_Alpha(src string, dedupOn string, destTable *bigquery.Table) (*bigquery.JobStatus, error)
  • func (dsExt *Dataset) DestQuery(query string, dest *bigquery.Table, disposition bigquery.TableWriteDisposition) *bigquery.Query
  • func (dsExt *Dataset) ExecDestQuery(q *bigquery.Query) (*bigquery.JobStatus, error)
  • func (dsExt Dataset) GetPartitionInfo(table string, partition string) (PartitionInfo, error)
  • func (dsExt *Dataset) QueryAndParse(q string, structPtr interface{}) error
  • func (dsExt *Dataset) ResultQuery(query string, dryRun bool) *bigquery.Query
• type PartitionInfo

Types

type Dataset

type Dataset struct {
	*bigquery.Dataset // Exposes Dataset API directly.
	BqClient          *bigquery.Client
}

Dataset provides extensions to the bigquery Dataset and Dataset objects to streamline common actions. It encapsulates the Client and Dataset to simplify methods.

func NewDataset

func NewDataset(project, dataset string, clientOpts ...option.ClientOption) (Dataset, error)

NewDataset creates a Dataset for a project. httpClient is used to inject mocks for the bigquery client. if httpClient is nil, a suitable default client is used. Additional bigquery ClientOptions may be optionally passed as final

clientOpts argument.  This is useful for testing credentials.

func (*Dataset) Dedup_Alpha

func (dsExt *Dataset) Dedup_Alpha(src string, dedupOn string, destTable *bigquery.Table) (*bigquery.JobStatus, error)

Dedup_Alpha executes a query that dedups and writes to destination partition. This function is alpha status. The interface may change without notice or major version number change.

`src` is relative to the project:dataset of dsExt. `dedupOn` names the field to be used for dedupping. `destTable` specifies the table to write to, typically created with

dsExt.BqClient.DatasetInProject(...).Table(...)

NOTE: If destination table is partitioned, destTable MUST include the partition suffix to avoid accidentally overwriting the entire table.

func (*Dataset) DestQuery

func (dsExt *Dataset) DestQuery(query string, dest *bigquery.Table, disposition bigquery.TableWriteDisposition) *bigquery.Query

DestinationQuery constructs a query with common Config settings for writing results to a table. If dest is nil, then this will create a DryRun query. TODO - should disposition be an opts... field instead?

func (*Dataset) ExecDestQuery

func (dsExt *Dataset) ExecDestQuery(q *bigquery.Query) (*bigquery.JobStatus, error)

ExecDestQuery executes a destination or dryrun query, and returns status or error.

func (Dataset) GetPartitionInfo

func (dsExt Dataset) GetPartitionInfo(table string, partition string) (PartitionInfo, error)

GetPartitionInfo provides basic information about a partition.

func (*Dataset) QueryAndParse

func (dsExt *Dataset) QueryAndParse(q string, structPtr interface{}) error

QueryAndParse executes a query that should return a single row, with all struct fields that match query columns filled in. The caller must pass in the *address* of an appropriate struct. TODO - extend this to also handle multirow results, by passing slice of structs.

func (*Dataset) ResultQuery

func (dsExt *Dataset) ResultQuery(query string, dryRun bool) *bigquery.Query

ResultQuery constructs a query with common QueryConfig settings for writing results to a table. Generally, may need to change WriteDisposition.

type PartitionInfo

type PartitionInfo struct {
	PartitionID  string
	CreationTime time.Time
	LastModified time.Time
}

PartitionInfo provides basic information about a partition.