bqx

Documentation

Index

• Variables
• func Customize(schema bigquery.Schema, subs map[string]bigquery.FieldSchema) bigquery.Schema
• func CustomizeAppend(schema bigquery.Schema, additions map[string]*bigquery.FieldSchema) bigquery.Schema
• func PrettyPrint(schema bigquery.Schema, simplify bool) (string, error)
• func RemoveRequired(schema bigquery.Schema) bigquery.Schema
• func UpdateSchemaDescription(schema bigquery.Schema, docs SchemaDoc) error
• func WalkSchema(schema bigquery.Schema, ...) error
• type PDT
  • func ParsePDT(fq string) (PDT, error)
  • func (pdt PDT) CreateTable(ctx context.Context, client *bigquery.Client, schema bigquery.Schema, ...) error
  • func (pdt PDT) UpdateTable(ctx context.Context, client *bigquery.Client, schema bigquery.Schema) error
• type SchemaDoc
  • func NewSchemaDoc(docs []byte) SchemaDoc

Variables

var (
	ErrInvalidProjectName = errors.New("Invalid project name")
	ErrInvalidDatasetName = errors.New("Invalid dataset name")
	ErrInvalidTableName   = errors.New("Invalid table name")
	ErrInvalidFQTable     = errors.New("Invalid fully qualified table name")
)

These errors are self-explanatory.

Functions

func Customize

func Customize(schema bigquery.Schema, subs map[string]bigquery.FieldSchema) bigquery.Schema

Customize recursively traverses a schema, substituting any fields that have a matching name in the provided map.

func CustomizeAppend

func CustomizeAppend(schema bigquery.Schema, additions map[string]*bigquery.FieldSchema) bigquery.Schema

CustomizeAppend recursively traverses a schema, appending the bigquery.FieldSchema to existing fields matching a name in the provided map.

func PrettyPrint

func PrettyPrint(schema bigquery.Schema, simplify bool) (string, error)

PrettyPrint generates a formatted json representation of a Schema. It simplifies the schema by removing zero valued fields, and compacting each field record onto a single line. Intended for diagnostics and debugging. Not suitable for production use.

func RemoveRequired

func RemoveRequired(schema bigquery.Schema) bigquery.Schema

RemoveRequired recursively traverses a schema, setting Required to false in all fields that are not fundamentally required by BigQuery.

func UpdateSchemaDescription

func UpdateSchemaDescription(schema bigquery.Schema, docs SchemaDoc) error

UpdateSchemaDescription walks each field in the given schema and assigns the Description field in place using values found in the given SchemaDoc.

func WalkSchema

func WalkSchema(schema bigquery.Schema, visit func(prefix []string, field *bigquery.FieldSchema) error) error

WalkSchema visits every FieldSchema object in the given schema by calling the visit function. The prefix is a path of field names from the top level to the current Field.

Types

type PDT

type PDT struct {
	Project string
	Dataset string
	Table   string
}

PDT contains a bigquery project, dataset, and table name.

func ParsePDT

func ParsePDT(fq string) (PDT, error)

ParsePDT parses and validates a fully qualified bigquery table name of the form project.dataset.table. None of the elements needs to exist, but all must conform to the corresponding naming restrictions.

func (PDT) CreateTable

func (pdt PDT) CreateTable(ctx context.Context, client *bigquery.Client, schema bigquery.Schema, description string,
	partitioning *bigquery.TimePartitioning, clustering *bigquery.Clustering) error

CreateTable will create a new table, or fail if the table already exists. It will also set appropriate time-partitioning field and clustering fields if non-nil arguments are provided. Returns error if the dataset does not already exist, or if other errors are encountered.

func (PDT) UpdateTable

func (pdt PDT) UpdateTable(ctx context.Context, client *bigquery.Client, schema bigquery.Schema) error

UpdateTable will update an existing table. Returns error if the table doesn't already exist, or if the schema changes are incompatible.

type SchemaDoc

type SchemaDoc map[string]map[string]string

SchemaDoc contains bigquery.Schema field Descriptions as read from an auxiliary source, such as YAML.

func NewSchemaDoc

func NewSchemaDoc(docs []byte) SchemaDoc

NewSchemaDoc reads the given file and attempts to parse it as a SchemaDoc. Errors are fatal.