dataset

README

The dataset package

This package contains extensions for the bigquery Dataset, to facilitate various common operations, notably query processing. It uses bqiface where possible to allow test fakes.

Documentation

Overview

Package x_dataset extends bqiface.Dataset with useful abstractions for simplifying interactions with bigquery. This package is intended to replace bqext. It is currently untested and buggy!!

Index

• Variables
• type Dataset
  • func NewDataset(ctx context.Context, project, dataset string, ...) (Dataset, error)
  • func (dsExt *Dataset) DestQuery(query string, dest bqiface.Table, disposition bigquery.TableWriteDisposition) bqiface.Query
  • func (dsExt Dataset) GetPartitionInfo(ctx context.Context, table string, partition string) (PartitionInfo, error)
  • func (dsExt *Dataset) QueryAndParse(ctx context.Context, q string, structPtr interface{}) error
  • func (dsExt *Dataset) ResultQuery(query string, dryRun bool) (bqiface.Query, error)
• type PartitionInfo

Variables

var (
	ErrNilBqClient = errors.New("nil BqClient")
	ErrNilQuery    = errors.New("BqClient.Query failed")
)

Errors returned by Dataset functions.

Types

type Dataset

type Dataset struct {
	bqiface.Dataset // Exposes Dataset API directly.
	BqClient        bqiface.Client
}

Dataset provides extensions to the bigquery Dataset and Dataset objects to streamline common actions. It encapsulates the Client and Dataset to simplify methods.

func NewDataset

func NewDataset(ctx context.Context, project, dataset string, clientOpts ...option.ClientOption) (Dataset, error)

NewDataset creates a Dataset for a project. httpClient is used to inject mocks for the bigquery client. if httpClient is nil, a suitable default client is used. Additional bigquery ClientOptions may be optionally passed as final

clientOpts argument.  This is useful for testing credentials.

NOTE: Caller should close the BqClient when finished.

func (*Dataset) DestQuery

func (dsExt *Dataset) DestQuery(query string, dest bqiface.Table, disposition bigquery.TableWriteDisposition) bqiface.Query

DestQuery constructs a query with common Config settings for writing results to a table. If dest is nil, then this will create a DryRun query. TODO - should disposition be an opts... field instead?

func (Dataset) GetPartitionInfo

func (dsExt Dataset) GetPartitionInfo(ctx context.Context, table string, partition string) (PartitionInfo, error)

GetPartitionInfo provides basic information about a partition.

func (*Dataset) QueryAndParse

func (dsExt *Dataset) QueryAndParse(ctx context.Context, q string, structPtr interface{}) error

QueryAndParse executes a query that should return a single row, with all struct fields that match query columns filled in. The caller must pass in the *address* of an appropriate struct. TODO - extend this to also handle multirow results, by passing slice of structs.

func (*Dataset) ResultQuery

func (dsExt *Dataset) ResultQuery(query string, dryRun bool) (bqiface.Query, error)

ResultQuery constructs a query with common QueryConfig settings for writing results to a table. Generally, may need to change WriteDisposition.

type PartitionInfo

type PartitionInfo struct {
	PartitionID  string
	CreationTime time.Time
	LastModified time.Time
}

PartitionInfo provides basic information about a partition.