control

Documentation

Overview

Package control provides methods to read and write records used to:

1. Ensure exactly-once ingestion of test results from builds.
2. Synchronise build completion and presubmit run completion, so that ingestion only proceeds when both build and presubmit run have completed.

Index

• Constants
• func BuildInvocationName(buildID int64) string
• func InsertOrUpdate(ctx context.Context, e *Entry) error
• func ReadBuildToInvocationJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
• func ReadBuildToPresubmitRunJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
• func ReadInvocationToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
• func ReadPresubmitToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)
• func SetEntriesForTesting(ctx context.Context, t testing.TB, es ...*Entry) (time.Time, error)
• func ValidateBuildResult(r *ctlpb.BuildResult) error
• func ValidateInvocationResult(r *ctlpb.InvocationResult) error
• func ValidatePresubmitResult(r *ctlpb.PresubmitResult) error
• type Entry
  • func Read(ctx context.Context, ingestionIDs []IngestionID) ([]*Entry, error)
• type EntryBuilder
  • func NewEntry(uniqifier int) *EntryBuilder
  • func (b *EntryBuilder) Build() *Entry
  • func (b *EntryBuilder) WithBuildJoinedTime(value time.Time) *EntryBuilder
  • func (b *EntryBuilder) WithBuildProject(project string) *EntryBuilder
  • func (b *EntryBuilder) WithBuildResult(value *controlpb.BuildResult) *EntryBuilder
  • func (b *EntryBuilder) WithHasBuildbucketBuild(hasBuildBucketBuild bool) *EntryBuilder
  • func (b *EntryBuilder) WithHasInvocation(hasInvocation bool) *EntryBuilder
  • func (b *EntryBuilder) WithIngestionID(id IngestionID) *EntryBuilder
  • func (b *EntryBuilder) WithInvocationJoinedTime(value time.Time) *EntryBuilder
  • func (b *EntryBuilder) WithInvocationProject(project string) *EntryBuilder
  • func (b *EntryBuilder) WithInvocationResult(value *controlpb.InvocationResult) *EntryBuilder
  • func (b *EntryBuilder) WithIsPresubmit(isPresubmit bool) *EntryBuilder
  • func (b *EntryBuilder) WithPresubmitJoinedTime(lastUpdated time.Time) *EntryBuilder
  • func (b *EntryBuilder) WithPresubmitProject(project string) *EntryBuilder
  • func (b *EntryBuilder) WithPresubmitResult(value *controlpb.PresubmitResult) *EntryBuilder
• type IngestionID
  • func IngestionIDFromBuildID(rdbHost string, buildID int64) IngestionID
  • func IngestionIDFromInvocationID(rdbHost, invocationID string) IngestionID
• type JoinStatistics

Constants

const JoinStatsHours = 36

JoinStatsHours is the number of previous hours ReadPresubmitRunJoinStatistics/ReadBuildJoinStatistics reads statistics for.

Functions

func BuildInvocationName

func BuildInvocationName(buildID int64) string

BuildInvocationName returns the invocation name corresponding to a buildbucket build. The pattern is originally defined here: https://source.chromium.org/chromium/infra/infra/+/main:appengine/cr-buildbucket/resultdb.py;l=75?q=build-%20resultdb&type=cs

func InsertOrUpdate

func InsertOrUpdate(ctx context.Context, e *Entry) error

InsertOrUpdate creates or updates the given ingestion record. This operation is not safe to perform blindly; perform only in a read/write transaction with an attempted read of the corresponding entry.

func ReadBuildToInvocationJoinStatistics

func ReadBuildToInvocationJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadBuildToInvocationJoinStatistics measures the performance joining builds to finalized invocations.

The statistics returned uses completed builds with an invocation as the denominator for measuring join performance. The performance joining to finalized invocations is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.

func ReadBuildToPresubmitRunJoinStatistics

func ReadBuildToPresubmitRunJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadBuildToPresubmitRunJoinStatistics measures the performance joining builds to presubmit runs.

The statistics returned uses completed builds with a presubmit run as the denominator for measuring join performance. The performance joining to presubmit run results is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.

func ReadInvocationToBuildJoinStatistics

func ReadInvocationToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadInvocationToBuildJoinStatistics measures the performance joining finalized invocations to builds.

The statistics returned uses finalized invocations (for buildbucket builds) as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the ingested invocation (this should be the same as the ingested build, although it comes from a different source). The last 36 hours of data for each project is returned. Hours are measured since the finalized invocation was received.

func ReadPresubmitToBuildJoinStatistics

func ReadPresubmitToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadPresubmitToBuildJoinStatistics measures the performance joining presubmit runs to builds.

The statistics returned uses builds as reported by completed presubmit runs as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the presubmit run. The last 36 hours of data for each project is returned. Hours are measured since the presubmit run result was received.

func SetEntriesForTesting

func SetEntriesForTesting(ctx context.Context, t testing.TB, es ...*Entry) (time.Time, error)

SetEntriesForTesting replaces the set of stored entries to match the given set.

func ValidateBuildResult

func ValidateBuildResult(r *ctlpb.BuildResult) error

func ValidateInvocationResult

func ValidateInvocationResult(r *ctlpb.InvocationResult) error

func ValidatePresubmitResult

func ValidatePresubmitResult(r *ctlpb.PresubmitResult) error

Types

type Entry

type Entry struct {
	// IngestionID uniquely identifies an ingestion.
	// The IngestionID is of the format {RDB_HOST}/{INVOCATION_ID}.
	// When the ingestion doesn't have an invocation, the INVOCATION_ID part of the IngestionID is a logical invocation id derived from the build ID.
	IngestionID IngestionID

	// Whether the invocation is from an build bucket build.
	// If true, ingestion should wait for the build result to be
	// populated before commencing ingestion.
	// Value only populated once either BuildResult or InvocationResult populated
	HasBuildBucketBuild bool

	// Project is the LUCI Project the build belongs to. Used for
	// metrics monitoring join performance.
	// This should be the same as InvocationProject.
	BuildProject string

	// BuildResult is the result of the build bucket build, to be passed
	// to the result ingestion task. This is nil if the result is
	// not yet known.
	BuildResult *ctlpb.BuildResult

	// BuildJoinedTime is the Spanner commit time the build result was
	// populated. If the result has not yet been populated, this is the zero time.
	BuildJoinedTime time.Time

	// HasInvocation records wether the build has an associated (ResultDB)
	// invocation.
	// Value only populated once either BuildResult or InvocationResult populated.
	HasInvocation bool

	// Project is the LUCI Project the invocation belongs to. Used for
	// metrics monitoring join performance.
	// This should be the same as BuildProject.
	InvocationProject string

	// InvocationResult is the result of the invocation, to be passed
	// to the result ingestion task. This is nil if the result is
	// not yet known.
	InvocationResult *ctlpb.InvocationResult

	// InvocationJoinedTime is the Spanner commit time the invocation result
	// was populated. If the result has not yet been populated, this is the zero time.
	InvocationJoinedTime time.Time

	// IsPresubmit records whether the build is part of a presubmit run.
	// If true, ingestion should wait for the presubmit result to be
	// populated (in addition to the build result) before commencing
	// ingestion.
	// Value only populated once either BuildResult or PresubmitResult populated.
	IsPresubmit bool

	// PresubmitProject is the LUCI Project the presubmit run belongs to.
	// This may differ from the LUCI Project the build belongs to. Used for
	// metrics monitoring join performance.
	PresubmitProject string

	// PresubmitResult is result of the presubmit run, to be passed to the
	// result ingestion task. This is nil if the result is not yet known.
	PresubmitResult *ctlpb.PresubmitResult

	// PresubmitJoinedTime is the Spanner commit time the presubmit result was
	// populated. If the result has not yet been populated, this is the zero time.
	PresubmitJoinedTime time.Time

	// LastUpdated is the Spanner commit time the row was last updated.
	LastUpdated time.Time
}

Entry is an ingestion control record, used to de-duplicate build ingestions and synchronise them with presubmit results (if required). An ingestion might contain the following entities. * A buildbucket build (not exist if the ingestion has an ResultDB invocation which does not associate with a buildbucket build). * A ResultDB invocation (not exist if the ingestion has an buildbucket build which does not contain invocation). * A CV run (only exist when the ingestion has buildbucket build and the build is a presubmit build). The ingestion task is only scheduled when all entities that exists are received.

func Read

func Read(ctx context.Context, ingestionIDs []IngestionID) ([]*Entry, error)

Read reads ingestion control records for the specified ingestionIDs. Exactly one *Entry is returned for each ingestionID. The result entry at index i corresponds to the ingestionIDs[i]. If a record does not exist for the given ingestionID, an *Entry of nil is returned for that ingestionID.

type EntryBuilder

type EntryBuilder struct {
	// contains filtered or unexported fields
}

EntryBuilder provides methods to build ingestion control records.

func NewEntry

func NewEntry(uniqifier int) *EntryBuilder

NewEntry starts building a new Entry.

func (*EntryBuilder) Build

func (b *EntryBuilder) Build() *Entry

Build constructs the entry.

func (*EntryBuilder) WithBuildJoinedTime

func (b *EntryBuilder) WithBuildJoinedTime(value time.Time) *EntryBuilder

WithBuildJoinedTime specifies the time the build result was populated.

func (*EntryBuilder) WithBuildProject

func (b *EntryBuilder) WithBuildProject(project string) *EntryBuilder

WithBuildProject specifies the build project to use on the ingestion control record.

func (*EntryBuilder) WithBuildResult

func (b *EntryBuilder) WithBuildResult(value *controlpb.BuildResult) *EntryBuilder

WithBuildResult specifies the build result for the entry.

func (*EntryBuilder) WithHasBuildbucketBuild

func (b *EntryBuilder) WithHasBuildbucketBuild(hasBuildBucketBuild bool) *EntryBuilder

WithHasBuildbucketBuild specifies whether the build that is the subject of the ingestion has a buildbucket build.

func (*EntryBuilder) WithHasInvocation

func (b *EntryBuilder) WithHasInvocation(hasInvocation bool) *EntryBuilder

WithHasInvocation specifies whether the build that is the subject of the ingestion has a ResultDB invocation.

func (*EntryBuilder) WithIngestionID

func (b *EntryBuilder) WithIngestionID(id IngestionID) *EntryBuilder

WithIngestionID specifies the ingestion ID to use on the ingestion control record.

func (*EntryBuilder) WithInvocationJoinedTime

func (b *EntryBuilder) WithInvocationJoinedTime(value time.Time) *EntryBuilder

WithInvocationJoinedTime specifies the time the invocation result was populated.

func (*EntryBuilder) WithInvocationProject

func (b *EntryBuilder) WithInvocationProject(project string) *EntryBuilder

WithInvocationProject specifies the invocation project to use on the ingestion control record.

func (*EntryBuilder) WithInvocationResult

func (b *EntryBuilder) WithInvocationResult(value *controlpb.InvocationResult) *EntryBuilder

WithInvocationResult specifies the invocation result for the entry.

func (*EntryBuilder) WithIsPresubmit

func (b *EntryBuilder) WithIsPresubmit(isPresubmit bool) *EntryBuilder

WithIsPresubmit specifies whether the ingestion relates to a presubmit run.

func (*EntryBuilder) WithPresubmitJoinedTime

func (b *EntryBuilder) WithPresubmitJoinedTime(lastUpdated time.Time) *EntryBuilder

WithPresubmitJoinedTime specifies the time the presubmit result was populated.

func (*EntryBuilder) WithPresubmitProject

func (b *EntryBuilder) WithPresubmitProject(project string) *EntryBuilder

WithPresubmitProject specifies the presubmit project to use on the ingestion control record.

func (*EntryBuilder) WithPresubmitResult

func (b *EntryBuilder) WithPresubmitResult(value *controlpb.PresubmitResult) *EntryBuilder

WithPresubmitResult specifies the build result for the entry.

type IngestionID

type IngestionID string

IngestionID represents the join logic between invocation, build and cv runs. IngestionID can be derived from build ID or invocation ID. Entities with the same ingestionID belongs to the same ingestion.

func IngestionIDFromBuildID

func IngestionIDFromBuildID(rdbHost string, buildID int64) IngestionID

IngestionIDFromBuildID returns the ingestionID for a given build id.

func IngestionIDFromInvocationID

func IngestionIDFromInvocationID(rdbHost, invocationID string) IngestionID

IngestionIDFromInvocationID returns the ingestionID for a given invocation id.

type JoinStatistics

type JoinStatistics struct {
	// TotalByHour captures the number of builds in the ingestionJoins
	// table eligible to be joined (i.e. have the left-hand join input).
	//
	// Data is broken down by by hours since the build became
	// eligible for joining. Index 0 indicates the period
	// from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on.
	TotalByHour []int64

	// JoinedByHour captures the number of builds in the ingestionJoins
	// table eligible to be joined, which were successfully joined (have
	// results for both join inputs present).
	//
	// Data is broken down by by hours since the build became
	// eligible for joining. Index 0 indicates the period
	// from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on.
	JoinedByHour []int64
}

JoinStatistics captures indicators of how well two join inputs (e.g. buildbucket build completions and presubmit run completions, or buildbucket build completions and invocation finalizations) are being joined.