bigquery

Documentation

Overview

Implements a sink to Google BigQuery, creating a dataset containing raw tables and de-duplicated views. The resulting dataset should be queryable as if it were the original Postgres database, and should support most queries without change.

Index

• func New(ctx context.Context, decoder decode.Decoder, opts Options) (generic.Sink, error)
• type Options
  • func (opt *Options) Bind(cmd *kingpin.CmdClause, prefix string) *Options
• type Table
  • func (i *Table) Insert(ctx context.Context, modifications []*changelog.Modification) (int, *uint64, error)

Functions

func New

func New(ctx context.Context, decoder decode.Decoder, opts Options) (generic.Sink, error)

Types

type Options

type Options struct {
	ProjectID     string
	Dataset       string
	Location      string
	BufferSize    int
	Instrument    bool
	FlushInterval time.Duration
}

func (*Options) Bind

func (opt *Options) Bind(cmd *kingpin.CmdClause, prefix string) *Options

type Table

type Table struct {
	Raw         *bq.Table
	RawMetadata *bq.TableMetadata
	Schema      *changelog.Schema
}

Table is a cached BigQuery table entry, associated with a schema. This is the table we insert modifications into.

func (*Table) Insert

func (i *Table) Insert(ctx context.Context, modifications []*changelog.Modification) (int, *uint64, error)

Insert bulk inserts changelog modifications into a BigQuery table. It returns the last non-nil LSN from the batch of modifications.