bigquery

Documentation

Index

• Constants
• func LogRecordsToBigQuery(bqSpec *BQSpec, items <-chan *bigquerytranslator.Item, successful *int32, ...)
• func NewInserter(ctx context.Context, resourceSpec, defaultProject string, ...) (inserter *bigquery.Inserter, cleanup func() error, err error)
• type BQClassifier
  • func (b BQClassifier) Classify(err error) retrier.Action
• type BQSpec
  • func (b *BQSpec) GetPeakWorkers() int32

Constants

const MaxUploadSize = 8 * 1000 * 1000 // slightly less than 10MB to compensate overhead.

Functions

func LogRecordsToBigQuery

func LogRecordsToBigQuery(bqSpec *BQSpec, items <-chan *bigquerytranslator.Item, successful *int32, failed *int32)

LogRecordsToBigQuery batches LogRecords, and uploads each batch to bigquery.

func NewInserter

func NewInserter(ctx context.Context, resourceSpec, defaultProject string, ts *oauth.TokenSource) (inserter *bigquery.Inserter, cleanup func() error, err error)

NewInserter creates an inserter for the table specified by the given resourceSpec in the form <project>:<dataset>.<table> or <dataset>.<table>

Types

type BQClassifier

type BQClassifier struct {
	// contains filtered or unexported fields
}

BQClassifier can classify bigquery related Errors for a Retrier. This Classifier interface is defined in https://github.com/eapache/go-resiliency/blob/main/retrier/classifier.go#L15

func (BQClassifier) Classify

func (b BQClassifier) Classify(err error) retrier.Action

Classify makes BQClassifier implements the Classifier interface, it classifies 400-level error codes as non-retryable.

type BQSpec

type BQSpec struct {
	Err         atomic.Pointer[error]
	ProjectID   string
	TableSpec   string
	BatchSizeMB int
	Client      *bigquery.Inserter
	CleanUp     func() error
	Ctx         context.Context
	Cancel      context.CancelFunc
	// contains filtered or unexported fields
}

BQSpec defines which bigquery table the LogRecords will be saved.

func (*BQSpec) GetPeakWorkers

func (b *BQSpec) GetPeakWorkers() int32