exporter

Documentation

Index

• type Formatter
• type JSONExporter
  • func New(bqClient bqiface.Client, projectID string, output Writer, format Formatter) *JSONExporter
  • func (exporter *JSONExporter) Export(ctx context.Context, config config.Config, queryTpl *template.Template, ...) error
• type QueryJob
• type UploadJob
• type UploadResult
• type Writer

Types

type Formatter

type Formatter interface {
	Source(project string, config config.Config, year int) string
	Partitions(source string) string
	Partition(row map[string]bigquery.Value) string
	Marshal(rows []map[string]bigquery.Value) ([]byte, error)
}

Formatter is the interface for all types that format table sources and queries used during the export process.

type JSONExporter

type JSONExporter struct {
	// contains filtered or unexported fields
}

JSONExporter is a JSON exporter for histogram data on BigQuery.

func New

func New(bqClient bqiface.Client, projectID string, output Writer, format Formatter) *JSONExporter

New creates a new JSONExporter.

func (*JSONExporter) Export

func (exporter *JSONExporter) Export(ctx context.Context,
	config config.Config, queryTpl *template.Template,
	year int) error

Export runs the provided SQL query and, for each row in the result, uploads a file to the provided config.OutputPath on GCS. This file contains the JSON representation of the rows. config.OutputPath is a template whose parameters are provided by the BigQuery row's fields. If a field is present in the output path, it's removed from the BigQuery row's JSON representation, to reduce redundancy.

e.g. if outputPath is "{{ .year }}/output.json" and we have a row per year, the histograms will be uploaded to: - 2010/output.json - 2020/output.json - etc.

If any of the steps (running the query, reading the result, marshalling, uploading) fails, this function returns the corresponding error.

Note: config.OutputPath should not start with a "/".

type QueryJob

type QueryJob struct {
	// contains filtered or unexported fields
}

QueryJob is a job for running queries on BQ.

type UploadJob

type UploadJob struct {
	// contains filtered or unexported fields
}

UploadJob is a job for uploading data to a GCS bucket.

type UploadResult

type UploadResult struct {
	// contains filtered or unexported fields
}

UploadResult is the result of a GCS upload.

type Writer

type Writer interface {
	Write(ctx context.Context, path string, content []byte) error
}

Writer defines the interface for saving files to GCS or locally.