bqlog

package
v0.0.0-...-d3da680 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 4, 2024 License: Apache-2.0 Imports: 32 Imported by: 2

Documentation

Overview

Package bqlog implements best effort low-overhead structured logging to BigQuery.

It is suitable only for data that can have occasional gaps or duplications, like debug logs. Do not use it for business-critical data.

Index

Constants

This section is empty.

Variables

View Source
var ModuleName = module.RegisterName("go.chromium.org/luci/server/bqlog")

ModuleName can be used to refer to this module when declaring dependencies.

Functions

func Log

func Log(ctx context.Context, m proto.Message)

Log asynchronously logs the given message to a BQ table associated with the message type via a prior RegisterSink call.

This is a best effort operation (and thus returns no error).

Messages are dropped when:

  • Writes to BigQuery are failing with a fatal error:
  • The table doesn't exist.
  • The table has an incompatible schema.
  • The server account has no permission to write to the table.
  • Etc.
  • The server crashes before it manages to flush buffered logs.
  • The internal flush buffer is full (per MaxLiveSizeBytes).

In case of transient write errors messages may be duplicated.

Panics if `m` was not registered via RegisterSink or if the bundler is not running yet.

func NewModule

func NewModule(opts *ModuleOptions) module.Module

NewModule returns a server module that sets up a cron dispatcher.

func NewModuleFromFlags

func NewModuleFromFlags() module.Module

NewModuleFromFlags is a variant of NewModule that initializes options through command line flags.

Calling this function registers flags in flag.CommandLine. They are usually parsed in server.Main(...).

func RegisterSink

func RegisterSink(sink Sink)

RegisterSink tells the bundler where and how to log messages of some concrete proto type.

There can currently be only one sink per proto message type. Must be called before the bundler is running. Can be called during the init() time.

Types

type BigQueryWriter

type BigQueryWriter interface {
	AppendRows(ctx context.Context, opts ...gax.CallOption) (storagepb.BigQueryWrite_AppendRowsClient, error)
	Close() error
}

BigQueryWriter is a subset of BigQueryWriteClient API used by Bundler.

Use storage.NewBigQueryWriteClient to construct a production instance.

type Bundler

type Bundler struct {
	CloudProject string // the cloud project with the dataset, required
	Dataset      string // the BQ dataset with log tables, required
	// contains filtered or unexported fields
}

Bundler buffers logs in memory before sending them to BigQuery.

var Default Bundler

Default is a bundler installed into the server when using NewModule or NewModuleFromFlags.

The module takes care of configuring this bundler based on the server environment and module's options.

You still need to register your types in it using RegisterSink.

func (*Bundler) Log

func (b *Bundler) Log(ctx context.Context, m proto.Message)

Log asynchronously logs the given message to a BQ table associated with the message type via a prior RegisterSink call.

This is a best effort operation (and thus returns no error).

Messages are dropped when:

  • Writes to BigQuery are failing with a fatal error:
  • The table doesn't exist.
  • The table has an incompatible schema.
  • The server account has no permission to write to the table.
  • Etc.
  • The server crashes before it manages to flush buffered logs.
  • The internal flush buffer is full (per MaxLiveSizeBytes).

In case of transient write errors messages may be duplicated.

Panics if `m` was not registered via RegisterSink or if the bundler is not running yet.

func (*Bundler) RegisterSink

func (b *Bundler) RegisterSink(sink Sink)

RegisterSink tells the bundler where and how to log messages of some concrete proto type.

There can currently be only one sink per proto message type. Must be called before the bundler is running. Can be called during the init() time.

func (*Bundler) Shutdown

func (b *Bundler) Shutdown(ctx context.Context)

Shutdown flushes pending logs and closes streaming RPCs.

Does nothing if the bundler wasn't running. Gives up waiting for all data to be flushed (and drops it) after 15s timeout or when `ctx` is canceled.

func (*Bundler) Start

func (b *Bundler) Start(ctx context.Context, w BigQueryWriter)

Start launches the bundler internal goroutines.

Canceling the context with cease all bundler activities. To gracefully shutdown the bundler (e.g. by flushing all pending logs) use Shutdown.

type FakeBigQueryWriter

type FakeBigQueryWriter struct {
	Send func(*storagepb.AppendRowsRequest) error
	Recv func() (*storagepb.AppendRowsResponse, error)
}

FakeBigQueryWriter is a fake for BigQueryWriter.

Calls given callbacks if they are non-nil. Useful in tests.

func (*FakeBigQueryWriter) AppendRows

func (*FakeBigQueryWriter) Close

func (*FakeBigQueryWriter) Close() error

type ModuleOptions

type ModuleOptions struct {
	// Bundler is a bundler to use.
	//
	// Default is the global Default instance.
	Bundler *Bundler

	// CloudProject is a project with the dataset to send logs to.
	//
	// Default is the project the server is running in.
	CloudProject string

	// Dataset is a BQ dataset to send logs to.
	//
	// Can be omitted in the development mode. In that case BQ writes will be
	// disabled and the module will be running in the dry run mode. Required in
	// the production.
	Dataset string
}

ModuleOptions contain configuration of the bqlog server module.

It will be used to initialize Default bundler.

func (*ModuleOptions) Register

func (o *ModuleOptions) Register(f *flag.FlagSet)

Register registers the command line flags.

type Sink

type Sink struct {
	Prototype        proto.Message // used only for its type descriptor, required
	Table            string        // the BQ table name within the bundler's dataset, required
	BatchAgeMax      time.Duration // for how long to buffer messages (or 0 for some default)
	MaxLiveSizeBytes int           // approximate limit on memory buffer size (or 0 for some default)
}

Sink describes where and how to log proto messages of the given type.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL