bq

Documentation

Overview

Package bq is responsible for all interactions with BigQuery.

Note that it uses the low level google.golang.org/api/bigquery/v2 for reasons that may longer be valid, but at the time this was written certain functionality was not supported by the higher level cloud.google.com/go/bigquery, specifically it had to do with how to create a client without a GOOGLE_APPLICATION_CREDENTIALS environment variable, see newBqApiClient().

Index

• func ParseTableSpec(spec string) (proj, ds, table string)
• type BigQuery
  • func NewBigQuery(cfg *Config) *BigQuery
  • func (b *BigQuery) GetJob(jobId string) (*bigquery.Job, error)
  • func (b *BigQuery) GetTable(dataset, table string) (*bigquery.Table, error)
  • func (b *BigQuery) NewExtractJobConfiguration(dataset, table string) *bigquery.JobConfiguration
  • func (b *BigQuery) NewLoadJobConfiguration(table, dataset, wdisp string, gsUrls []string, format string) *bigquery.JobConfiguration
  • func (b *BigQuery) NewQueryJobConfiguration(sql, dataset, table, disp string, legacy, partitioned bool) *bigquery.JobConfiguration
  • func (b *BigQuery) ProjectId() string
  • func (b *BigQuery) SignedStorageUrl(filename, method string) (string, error)
  • func (b *BigQuery) StartJob(conf *bigquery.JobConfiguration) (*bigquery.Job, error)
  • func (b *BigQuery) TableColumnNames(dataset, table string) ([]string, error)
  • func (b *BigQuery) TableData(dataset, table string, f func([]string) error) error
• type Config

Functions

func ParseTableSpec

func ParseTableSpec(spec string) (proj, ds, table string)

Parse a qualified BigQuery table name, such as "[orgname:project.table]" (Legacy) or `orgname.project.table` (Standard). Returns three strings: project, dataset and table. (Blank if not specified).

Types

type BigQuery

type BigQuery struct {
	// contains filtered or unexported fields
}

BigQuery is a struct containing the information necessary to communicate with the BigQuery API.

func NewBigQuery

func NewBigQuery(cfg *Config) *BigQuery

NewBigQuery returns a BigQuery instance given a Config pointer.

func (*BigQuery) GetJob

func (b *BigQuery) GetJob(jobId string) (*bigquery.Job, error)

Retrieve Job information from the BigQuery API given a JobId.

func (*BigQuery) GetTable

func (b *BigQuery) GetTable(dataset, table string) (*bigquery.Table, error)

Retrieve BigQuery table information form the BigQuery API given a dataset and a table.

func (*BigQuery) NewExtractJobConfiguration

func (b *BigQuery) NewExtractJobConfiguration(dataset, table string) *bigquery.JobConfiguration

Create an extract JobConfiguration to be able to start an extract job.

func (*BigQuery) NewLoadJobConfiguration

func (b *BigQuery) NewLoadJobConfiguration(table, dataset, wdisp string, gsUrls []string, format string) *bigquery.JobConfiguration

func (*BigQuery) NewQueryJobConfiguration

func (b *BigQuery) NewQueryJobConfiguration(sql, dataset, table, disp string, legacy, partitioned bool) *bigquery.JobConfiguration

Create a query JobConfiguration given the SQL, the destination dataset and table, disposition ("WRITE_APPEND" or "WRITE_TRUNCATE), whether legacy SQL is used and whether the resulting table is DAY partitioned."

func (*BigQuery) ProjectId

func (b *BigQuery) ProjectId() string

func (*BigQuery) SignedStorageUrl

func (b *BigQuery) SignedStorageUrl(filename, method string) (string, error)

Construct and return a cryptographically signed URL to a GCS file. This URL can be used to retrieve the file without any authentication. The signature expires after duration provided to NewBigQuery().

func (*BigQuery) StartJob

func (b *BigQuery) StartJob(conf *bigquery.JobConfiguration) (*bigquery.Job, error)

Start a BigQuery job given a JobConfiguration.

func (*BigQuery) TableColumnNames

func (b *BigQuery) TableColumnNames(dataset, table string) ([]string, error)

Get table column names as a []string

func (*BigQuery) TableData

func (b *BigQuery) TableData(dataset, table string, f func([]string) error) error

Given a function which accepts a []string, call it repeatdly for every row of data. This is meant for tables that do not have repeated fields, but if a repeated field is encountered, it will be simply marshalled as JSON. This uses the BigQuery paging API which is slow and only suitable for small tables.

type Config

type Config struct {
	ProjectId     string        // ProjectId
	Email         string        // Email (for authentication)
	Key           string        // Key (for authentication)
	GcsBucket     string        // Bucket used for GCS exports
	GcsExpiration time.Duration // Expire GCS URL after (defaults to 4 hours)
}

Configration parameters to instantiate a BigQuery connection.