bqwrapper

Documentation

Overview

Package bqwrapper is a wrapper around the bigquery API.

It does some things a little differently. The real bigquery go library uses a fluent interface. This library does not.

It does not expose very much (only what is necessary for the projects that are using it). If you need more functionality, please send a CL.

Anyway, the wrapper interface has two implementations:

1) CloudBQ -- the production one. 2) MemBQ -- the testing one, which is implemented on top of the datastore emulator that we already have.

I have made some attempt to mimic the semantics of the real BigQuery, but I didn't try very hard. This package is mostly there to allow unit code that uses bigquery to be unit-tested in a reasonable way.

Index

• type BQIf
• type CloudBQ
  • func NewCloudBQ(client *bigquery.Client) *CloudBQ
  • func (cbq *CloudBQ) Put(ctx context.Context, projectID string, dataset string, table string, ...) error
• type EmulatedBigqueryRecord
• type MemBQ
  • func MakeMemBQ(ctx context.Context) (*MemBQ, error)
  • func (mbq *MemBQ) Put(ctx context.Context, projectID string, dataset string, table string, ...) error
  • func (mbq *MemBQ) UniversalRowQuery(ctx context.Context) *datastore.Query

Types

type BQIf

type BQIf interface {
	// Put inserts some records into Bigquery.
	//
	// Note that you always specify the projectID, dataset, and table.
	//
	// The Save method on each element of data MUST expose datastore-compatible types.
	//
	// The exact rules for how a go type gets converted to datastore by this library will
	// probably change in the future, depending on what exactly we try to store in BigQuery
	// in practice.
	Put(ctx context.Context, projectID string, dataset string, table string, data []bigquery.ValueSaver) error
}

BQIf is the bigquery wrapper interface.

It exposes the subset of the bigquery interface that can be used while testing and in prod.

type CloudBQ

type CloudBQ struct {
	// contains filtered or unexported fields
}

CloudBQ wraps the prod client.

func NewCloudBQ

func NewCloudBQ(client *bigquery.Client) *CloudBQ

NewCloudBQ makes a new one.

func (*CloudBQ) Put

func (cbq *CloudBQ) Put(ctx context.Context, projectID string, dataset string, table string, data []bigquery.ValueSaver) error

Put writes a record to BigQuery.

type EmulatedBigqueryRecord

type EmulatedBigqueryRecord struct {

	// Make this public so unit tests can consume them.
	ProjectID string `gae:"____project_id"`
	Dataset   string `gae:"____dataset"`
	Table     string `gae:"____table"`
	// Extra *has* to be exported or the datastore ORM will crash.
	Extra datastore.PropertyMap `gae:",extra"`
	// contains filtered or unexported fields
}

EmulatedBigqueryRecord is a record in datastore that emulates a bigquery row.

Fields used internally by the emulator are prefixed with FOUR underscores. ____

type MemBQ

type MemBQ struct{}

func MakeMemBQ

func MakeMemBQ(ctx context.Context) (*MemBQ, error)

MakeMemBQ makes a new MemBQ.

func (*MemBQ) Put

func (mbq *MemBQ) Put(ctx context.Context, projectID string, dataset string, table string, data []bigquery.ValueSaver) error

Put writes a record to the fake datastore and thus the fake BigQuery.

func (*MemBQ) UniversalRowQuery

func (mbq *MemBQ) UniversalRowQuery(ctx context.Context) *datastore.Query

UniversalRowQuery gives a query over the row records.