bqext

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 13, 2017 License: Apache-2.0 Imports: 10 Imported by: 0

README

The bqext package

This package contains extensions for the bigquery library, to facilitate various common operations, notably query processing.

Documentation

Overview

Package bqext includes generally useful abstractions for simplifying interactions with bigquery. Production extensions should go here, but test facilities should go in a separate bqtest package.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Dataset added in v0.1.2

type Dataset struct {
	BqClient *bigquery.Client
	Dataset  *bigquery.Dataset
}

Dataset provides extensions to the bigquery Dataset and Dataset objects to streamline common actions. It encapsulates the Client and Dataset to simplify methods.

func NewDataset added in v0.1.2

func NewDataset(project, dataset string, clientOpts ...option.ClientOption) (Dataset, error)

NewDataset creates a Dataset for a project. httpClient is used to inject mocks for the bigquery client. if httpClient is nil, a suitable default client is used. Additional bigquery ClientOptions may be optionally passed as final

clientOpts argument.  This is useful for testing credentials.

func (*Dataset) Dedup added in v0.1.2

func (dsExt *Dataset) Dedup(src string, dedupOn string, overwrite bool, project, dataset, table string) (*bigquery.JobStatus, error)

Dedup executes a query that dedups and writes to an appropriate partition. src is relative to the project:dataset of dsExt. dedupOn names the field to be used for dedupping. project, dataset, table specify the table to write into. NOTE: destination table must include the partition suffix. This avoids accidentally overwriting TODAY's partition.

func (*Dataset) DestinationQuery added in v0.1.2

func (dsExt *Dataset) DestinationQuery(query string, dest *bigquery.Table) *bigquery.Query

DestinationQuery constructs a query with common Config settings for writing results to a table. Generally, may need to change WriteDisposition.

func (Dataset) GetPartitionInfo added in v0.1.2

func (dsExt Dataset) GetPartitionInfo(table string, partition string) (PartitionInfo, error)

GetPartitionInfo provides basic information about a partition.

func (*Dataset) QueryAndParse added in v0.1.2

func (dsExt *Dataset) QueryAndParse(q string, structPtr interface{}) error

QueryAndParse executes a query that should return a single row, with all struct fields that match query columns filled in. The caller must pass in the *address* of an appropriate struct. TODO - extend this to also handle multirow results, by passing slice of structs.

func (*Dataset) ResultQuery added in v0.1.2

func (dsExt *Dataset) ResultQuery(query string, dryRun bool) *bigquery.Query

ResultQuery constructs a query with common QueryConfig settings for writing results to a table. Generally, may need to change WriteDisposition.

type PartitionInfo added in v0.1.2

type PartitionInfo struct {
	PartitionID  string
	CreationTime time.Time
	LastModified time.Time
}

PartitionInfo provides basic information about a partition.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL