bigquerysink

Documentation

Index

• Variables
• func MustNew(client *bigquery.Client, options ...Option) pipeline.Sink
• type Message
• type Option
  • func WithIgnoreUnkownMessageTypes() Option
  • func WithIgnoreValidationErrors() Option
  • func WithLogLevel(l zerolog.Level) Option

Variables

var (
	// ErrUnkownMessageType is returned when the sink message
	// received is not of the type Message.
	ErrUnkownMessageType = errors.New("unknown message type")

	// ErrEmptyProjectID is returned when the Message.ProjectID is empty.
	ErrEmptyProjectID = errors.New("project id is missing from the message")

	// ErrEmptyDatasetID is returned when the Message.DatasetID is empty.
	ErrEmptyDatasetID = errors.New("dataset id is missing from the message")

	// ErrEmptyTableID is returned when the Message.TableID is empty.
	ErrEmptyTableID = errors.New("table id is missing from the message")

	// ErrEmptyData is returned when the Message.Data is nil.
	ErrEmptyData = errors.New("data is missing from the message")
)

Functions

func MustNew

func MustNew(client *bigquery.Client, options ...Option) pipeline.Sink

MustNew returns a new Sink that saves data to bigquiery.

Types

type Message

type Message struct {
	// TableID is the table name where the Data will be saved.
	TableID string
	// DatasetID is the dataset containing the Table.
	DatasetID string
	// ProjectID is the GCP's project Id which contains the dataset.
	ProjectID string

	// Data is the data being saved. It should be a type compatible with bigquery package.
	// This means that types as `map` or  `*string` will fail.
	// Read the bigquery documentation for more details: https://pkg.go.dev/cloud.google.com/go/bigquery.
	Data interface{}
}

Message is the pipeline message being stored.

type Option

type Option func(*bigquerySink)

Option configures the bigquery sink.

func WithIgnoreUnkownMessageTypes

func WithIgnoreUnkownMessageTypes() Option

WithIgnoreUnkownMessageTypes will make the bigquery sink ignore messages that are not of type Message. The default behavior is return an error if the message is not of the expected type.

func WithIgnoreValidationErrors

func WithIgnoreValidationErrors() Option

WithIgnoreValidationErrors will make the bigquery sink ignore incomplete messages. Otherwise it will return an error.

func WithLogLevel

func WithLogLevel(l zerolog.Level) Option

WithLogLevel changes the bigquery allowed log level. The bigquery sink will emit log messages bellow this level. The default value is zerolog.Disabled. No log is emitted. Setting it to zerolog.Trace will enable all messages.