artifactexporter

Documentation

Overview

Package artifactexporter handles uploading artifacts to BigQuery. This is the replacement of the legacy artifact exporter in bqexp

Index

• Constants
• Variables
• func InitServer(srv *server.Server, opts Options) error
• func Schedule(ctx context.Context, invID invocations.ID) error
• type Artifact
• type BQExportClient
• type Client
  • func NewClient(ctx context.Context, projectID string) (s *Client, reterr error)
  • func (c *Client) Close() (reterr error)
  • func (c *Client) EnsureSchema(ctx context.Context) error
  • func (c *Client) InsertArtifactRows(ctx context.Context, rows []*bqpb.TextArtifactRow) error
• type Options
• type Row

Constants

const ArtifactRequestOverhead = 100

ArtifactRequestOverhead is the overhead (in bytes) applying to each artifact when calculating the size.

const CheckpointProcessID = "artifact-exporter"

const CheckpointTTL = 7 * 24 * time.Hour

CheckpointTTL specifies the TTL for checkpoints in Spanner. 7 days for checkpoints should be enough to handle any production issues. Another option is to set it to 510 days (equals to Artifact TTL), but that may be wasteful.

const MaxArtifactSize = 100 * 1024 * 1024

MaxArtifactSize is the maximum size of an artifact to be exported to BigQuery. Artifacts bigger than this size will not be exported.

const MaxRBECasBatchSize = 2 * 1024 * 1024 // 2 MB

MaxRBECasBatchSize is the batch size limit when we read artifact content. TODO(nqmtuan): Call the Capacity API to find out the exact size limit for batch operations. For now, hardcode to be 2MB. It should be under the limit, since BatchUpdateBlobs in BatchCreateArtifacts can handle up to 10MB. If you need to change this value, please make sure MaxRBECasBatchSize < MaxShardContentSize, as we only use 1 Bigquery shard to store these small artifacts.

const MaxShardContentSize = bq.RowMaxBytes - 10*1024

MaxShardContentSize is the maximum content size in BQ row. Artifacts content bigger than this size needs to be sharded. Leave 10 KB for other fields, the rest is content. If you need to change this value, please be aware that it may result in a different way to split an artifact. This may lead to the current checkpoint being incorrect.

const MaxTotalArtifactSizeForInvocation = 5 * 1024 * 1024 * 1024 // 5GB

MaxTotalArtifactSizeForInvocation is the max total size of artifacts in an invocation. If an invocation has total artifact size exceeding this value, we will not export it. The reason is that the cloud task cannot finish the export within 10 minutes, leading to a timeout.

Variables

var (
	ErrInvalidUTF8 = fmt.Errorf("invalid UTF-8 character")
)

var ExportArtifactsTask = tq.RegisterTaskClass(tq.TaskClass{
	ID:            "export-artifacts",
	Prototype:     &taskspb.ExportArtifacts{},
	Kind:          tq.Transactional,
	Queue:         "artifactexporter",
	RoutingPrefix: "/internal/tasks/artifactexporter",
})

ExportArtifactsTask describes how to route export artifact task.

Functions

func InitServer

func InitServer(srv *server.Server, opts Options) error

InitServer initializes a artifactexporter server.

func Schedule

func Schedule(ctx context.Context, invID invocations.ID) error

Schedule schedules tasks for all the finalized invocations.

Types

type Artifact

type Artifact struct {
	InvocationID    string
	TestID          string
	ResultID        string
	ArtifactID      string
	ContentType     string
	Size            int64
	RBECASHash      string
	TestStatus      pb.TestStatus
	TestVariant     *pb.Variant
	TestVariantHash string
}

type BQExportClient

type BQExportClient interface {
	InsertArtifactRows(ctx context.Context, rows []*bqpb.TextArtifactRow) error
}

BQExportClient is the interface for exporting artifacts.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client provides methods to export artifacts to BigQuery via the BigQuery Write API.

func NewClient

func NewClient(ctx context.Context, projectID string) (s *Client, reterr error)

NewClient creates a new client for exporting artifacts via the BigQuery Write API.

func (*Client) Close

func (c *Client) Close() (reterr error)

Close releases resources held by the client.

func (*Client) EnsureSchema

func (c *Client) EnsureSchema(ctx context.Context) error

func (*Client) InsertArtifactRows

func (c *Client) InsertArtifactRows(ctx context.Context, rows []*bqpb.TextArtifactRow) error

InsertArtifactRows inserts the given rows in BigQuery.

type Options

type Options struct {
	// ArtifactRBEInstance is the name of the RBE instance to use for artifact
	// storage. Example: "projects/luci-resultdb/instances/artifacts".
	ArtifactRBEInstance string
}

Options is artifact exporter configuration.

type Row

type Row struct {
	// contains filtered or unexported fields
}

Row represent the data to be inserted in the exporter channel.