updater

Documentation

Overview

Package updater reads the latest test results and saves updated state.

Index

• Constants
• func AddManualSubscription(projID, subID, prefix string)
• func ConstructGrid(log logrus.FieldLogger, group *configpb.TestGroup, cols []InflatedColumn, ...) *statepb.Grid
• func InflateDropAppend(ctx context.Context, alog logrus.FieldLogger, client gcs.Client, ...) (bool, error)
• func Means(properties map[string][]string) map[string]float64
• func SortStarted(_ *configpb.TestGroup, cols []InflatedColumn)
• func SplitCells(originalName string, cells ...Cell) map[string]Cell
• func Update(parent context.Context, client gcs.ConditionalClient, mets *Metrics, ...) error
• type Cell
  • func MergeCells(flaky bool, cells ...Cell) Cell
• type ColumnReader
• type ColumnSorter
• type Fixer
  • func FixGCS(subscriber pubsub.Subscriber) Fixer
• type GroupUpdater
  • func GCS(colClient gcs.Client, groupTimeout, buildTimeout time.Duration, ...) GroupUpdater
• type InflatedColumn
  • func InflateGrid(grid *statepb.Grid, earliest, latest time.Time) ([]InflatedColumn, map[string][]string)
• type Metrics

Constants

const ElapsedKey = "test-duration-minutes"

ElapsedKey is the key for the test duration metric.

const EmailListKey = "EmailAddresses"

EmailListKey is the expected metadata key for email addresses.

Functions

func AddManualSubscription

func AddManualSubscription(projID, subID, prefix string)

AddManualSubscription allows injecting additional subscriptions that are not specified by the test group itself.

Likely to be removed (or migrated into the config.proto) in a future version.

func ConstructGrid

func ConstructGrid(log logrus.FieldLogger, group *configpb.TestGroup, cols []InflatedColumn, issues map[string][]string) *statepb.Grid

ConstructGrid will append all the inflatedColumns into the returned Grid.

The returned Grid has correctly compressed row values.

func InflateDropAppend

func InflateDropAppend(ctx context.Context, alog logrus.FieldLogger, client gcs.Client, tg *configpb.TestGroup, gridPath gcs.Path, write bool, readCols ColumnReader, sortCols ColumnSorter, reprocess time.Duration) (bool, error)

InflateDropAppend updates groups by downloading the existing grid, dropping old rows and appending new ones.

func Means

func Means(properties map[string][]string) map[string]float64

Means returns means for each given property's values.

func SortStarted

func SortStarted(_ *configpb.TestGroup, cols []InflatedColumn)

SortStarted sorts InflatedColumns by column start time.

func SplitCells

func SplitCells(originalName string, cells ...Cell) map[string]Cell

SplitCells appends a unique suffix to each cell.

When an excessive number of cells contain the same name the list gets truncated, replaced with a synthetic "... [overflow]" cell.

func Update

func Update(parent context.Context, client gcs.ConditionalClient, mets *Metrics, configPath gcs.Path, gridPrefix string, groupConcurrency int, groupNames []string, updateGroup GroupUpdater, write bool, freq time.Duration, fixers ...Fixer) error

Update test groups with the specified freq.

Retries errors at double and unfinished groups as soon as possible.

Filters down to a single group when set. Returns after all groups updated once if freq is zero.

Types

type Cell

type Cell struct {
	// Result determines the color of the cell, defaulting to NO_RESULT (clear)
	Result statuspb.TestStatus

	// The name of the row before user-customized formatting
	ID string

	// CellID specifies the an identifier to the build, which allows
	// clicking different cells in a column to go to different locations.
	CellID string

	// Properties maps key:value pairs for cell IDs.
	Properties map[string]string

	// Icon is a short string that appears on the cell
	Icon string
	// Message is a longer string that appears on mouse-over
	Message string

	// Metrics holds numerical data, such as how long it ran, coverage, etc.
	Metrics map[string]float64

	// UserProperty holds the value of a user-defined property, which allows
	// runtime flexibility in generating links to click on.
	UserProperty string

	// Issues relevant to this cell
	// TODO(fejta): persist cell association, currently gets written out as a row-association.
	// TODO(fejta): support issue association when parsing prow job results.
	Issues []string
}

Cell holds a row's values for a given column

func MergeCells

func MergeCells(flaky bool, cells ...Cell) Cell

MergeCells will combine the cells into a single result.

The flaky argument determines whether returned result is flaky (true) or failing when merging cells with both passing and failing results.

Merging multiple results will set the icon to n/N passes

Includes the message from the "most relevant" cell that includes a message. Where relevance is determined by result.GTE.

type ColumnReader

type ColumnReader func(ctx context.Context, log logrus.FieldLogger, tg *configpb.TestGroup, oldCols []InflatedColumn, stop time.Time, receivers chan<- InflatedColumn) error

ColumnReader finds, processes and new columns to send to the receivers.

* Columns with the same Name and Build will get merged together. * Readers must be reentrant.

• Processing must expect every sent column to be the final column this cycle. AKA calling this method once and reading two columns should be equivalent to calling the method once, reading one column and then calling it a second time and reading a second column.

type ColumnSorter

type ColumnSorter func(*configpb.TestGroup, []InflatedColumn)

A ColumnSorter sort InflatedColumns as desired.

type Fixer

type Fixer func(context.Context, logrus.FieldLogger, *config.TestGroupQueue, []*configpb.TestGroup) error

Fixer will fix the TestGroupQueue's next time for TestGroups.

Fixer should: * work continually and not return until the context expires. * expect to be called multiple times with different contexts and test groups.

For example, it might use the last updated time of the test group to specify the next update time. Or it might watch the data backing these groups and request an immediate update whenever the data changes.

func FixGCS

func FixGCS(subscriber pubsub.Subscriber) Fixer

FixGCS listens for changes to GCS files and schedules another update of those groups ~immediately.

Limited to test groups with a gcs_config result_source that includes pubsub info. Returns when the context is canceled or a processing error occurs.

type GroupUpdater

type GroupUpdater func(parent context.Context, log logrus.FieldLogger, client gcs.Client, tg *configpb.TestGroup, gridPath gcs.Path) (bool, error)

GroupUpdater will compile the grid state proto for the specified group and upload it.

This typically involves downloading the existing state, dropping old columns, compiling any new columns and inserting them into the front and then uploading the proto to GCS.

Return true if there are more results to process.

func GCS

func GCS(colClient gcs.Client, groupTimeout, buildTimeout time.Duration, concurrency int, write bool, sortCols ColumnSorter) GroupUpdater

GCS returns a GCS-based GroupUpdater, which knows how to process result data stored in GCS.

type InflatedColumn

type InflatedColumn struct {
	// Column holds the header data.
	Column *statepb.Column
	// Cells holds each row's uncompressed data for this column.
	Cells map[string]Cell
}

InflatedColumn holds all the entries for a given column.

This includes both: * Column state metadata and * Cell values for every row in this column

func InflateGrid

func InflateGrid(grid *statepb.Grid, earliest, latest time.Time) ([]InflatedColumn, map[string][]string)

InflateGrid inflates the grid's rows into an InflatedColumn channel.

Drops columns before earliest or more recent than latest. Also returns a map of issues associated with each row name.

type Metrics

type Metrics struct {
	Errors       metrics.Counter
	Skips        metrics.Counter
	Successes    metrics.Counter
	DelaySeconds metrics.Int64
	CycleSeconds metrics.Int64
}

Metrics holds metrics relevant to the Updater.