control

package
v0.0.0-...-110cdec Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2024 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package control provides methods to read and write records used to:

  1. Ensure exactly-once ingestion of test results from builds.
  2. Synchronise build completion and presubmit run completion, so that ingestion only proceeds when both build and presubmit run have completed.

Index

Constants

View Source 
const JoinStatsHours = 36

JoinStatsHours is the number of previous hours ReadPresubmitRunJoinStatistics/ReadBuildJoinStatistics reads statistics for.

Variables

This section is empty.

Functions

func BuildInvocationName 

func BuildInvocationName(buildID int64) string

BuildInvocationName returns the invocation name corresponding to a buildbucket build. The pattern is originally defined here: https://source.chromium.org/chromium/infra/infra/+/main:appengine/cr-buildbucket/resultdb.py;l=75?q=build-%20resultdb&type=cs

func InsertOrUpdate 

func InsertOrUpdate(ctx context.Context, e *Entry) error

InsertOrUpdate creates or updates the given ingestion record. This operation is not safe to perform blindly; perform only in a read/write transaction with an attempted read of the corresponding entry.

func ReadBuildToInvocationJoinStatistics 

func ReadBuildToInvocationJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadBuildToInvocationJoinStatistics measures the performance joining builds to finalized invocations.

The statistics returned uses completed builds with an invocation as the denominator for measuring join performance. The performance joining to finalized invocations is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.

func ReadBuildToPresubmitRunJoinStatistics 

func ReadBuildToPresubmitRunJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadBuildToPresubmitRunJoinStatistics measures the performance joining builds to presubmit runs.

The statistics returned uses completed builds with a presubmit run as the denominator for measuring join performance. The performance joining to presubmit run results is then measured. Data is broken down by the project of the buildbucket build. The last 36 hours of data for each project is returned. Hours are measured since the buildbucket build result was received.

func ReadInvocationToBuildJoinStatistics 

func ReadInvocationToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadInvocationToBuildJoinStatistics measures the performance joining finalized invocations to builds.

The statistics returned uses finalized invocations (for buildbucket builds) as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the ingested invocation (this should be the same as the ingested build, although it comes from a different source). The last 36 hours of data for each project is returned. Hours are measured since the finalized invocation was received.

func ReadPresubmitToBuildJoinStatistics 

func ReadPresubmitToBuildJoinStatistics(ctx context.Context) (map[string]JoinStatistics, error)

ReadPresubmitToBuildJoinStatistics measures the performance joining presubmit runs to builds.

The statistics returned uses builds as reported by completed presubmit runs as the denominator for measuring join performance. The performance joining to buildbucket build results is then measured. Data is broken down by the project of the presubmit run. The last 36 hours of data for each project is returned. Hours are measured since the presubmit run result was received.

func SetEntriesForTesting 

func SetEntriesForTesting(ctx context.Context, t testing.TB, es ...*Entry) (time.Time, error)

SetEntriesForTesting replaces the set of stored entries to match the given set.

func ValidateBuildResult 

func ValidateBuildResult(r *ctlpb.BuildResult) error

func ValidateInvocationResult 

func ValidateInvocationResult(r *ctlpb.InvocationResult) error

func ValidatePresubmitResult 

func ValidatePresubmitResult(r *ctlpb.PresubmitResult) error

Types

type Entry 

type Entry struct {
	// IngestionID uniquely identifies an ingestion.
	// The IngestionID is of the format {RDB_HOST}/{INVOCATION_ID}.
	// When the ingestion doesn't have an invocation, the INVOCATION_ID part of the IngestionID is a logical invocation id derived from the build ID.
	IngestionID IngestionID

	// Whether the invocation is from an build bucket build.
	// If true, ingestion should wait for the build result to be
	// populated before commencing ingestion.
	// Value only populated once either BuildResult or InvocationResult populated
	HasBuildBucketBuild bool

	// Project is the LUCI Project the build belongs to. Used for
	// metrics monitoring join performance.
	// This should be the same as InvocationProject.
	BuildProject string

	// BuildResult is the result of the build bucket build, to be passed
	// to the result ingestion task. This is nil if the result is
	// not yet known.
	BuildResult *ctlpb.BuildResult

	// BuildJoinedTime is the Spanner commit time the build result was
	// populated. If the result has not yet been populated, this is the zero time.
	BuildJoinedTime time.Time

	// HasInvocation records wether the build has an associated (ResultDB)
	// invocation.
	// Value only populated once either BuildResult or InvocationResult populated.
	HasInvocation bool

	// Project is the LUCI Project the invocation belongs to. Used for
	// metrics monitoring join performance.
	// This should be the same as BuildProject.
	InvocationProject string

	// InvocationResult is the result of the invocation, to be passed
	// to the result ingestion task. This is nil if the result is
	// not yet known.
	InvocationResult *ctlpb.InvocationResult

	// InvocationJoinedTime is the Spanner commit time the invocation result
	// was populated. If the result has not yet been populated, this is the zero time.
	InvocationJoinedTime time.Time

	// IsPresubmit records whether the build is part of a presubmit run.
	// If true, ingestion should wait for the presubmit result to be
	// populated (in addition to the build result) before commencing
	// ingestion.
	// Value only populated once either BuildResult or PresubmitResult populated.
	IsPresubmit bool

	// PresubmitProject is the LUCI Project the presubmit run belongs to.
	// This may differ from the LUCI Project the build belongs to. Used for
	// metrics monitoring join performance.
	PresubmitProject string

	// PresubmitResult is result of the presubmit run, to be passed to the
	// result ingestion task. This is nil if the result is not yet known.
	PresubmitResult *ctlpb.PresubmitResult

	// PresubmitJoinedTime is the Spanner commit time the presubmit result was
	// populated. If the result has not yet been populated, this is the zero time.
	PresubmitJoinedTime time.Time

	// LastUpdated is the Spanner commit time the row was last updated.
	LastUpdated time.Time
}

Entry is an ingestion control record, used to de-duplicate build ingestions and synchronise them with presubmit results (if required). An ingestion might contain the following entities. * A buildbucket build (not exist if the ingestion has an ResultDB invocation which does not associate with a buildbucket build). * A ResultDB invocation (not exist if the ingestion has an buildbucket build which does not contain invocation). * A CV run (only exist when the ingestion has buildbucket build and the build is a presubmit build). The ingestion task is only scheduled when all entities that exists are received.

func Read 

func Read(ctx context.Context, ingestionIDs []IngestionID) ([]*Entry, error)

Read reads ingestion control records for the specified ingestionIDs. Exactly one *Entry is returned for each ingestionID. The result entry at index i corresponds to the ingestionIDs[i]. If a record does not exist for the given ingestionID, an *Entry of nil is returned for that ingestionID.

type EntryBuilder 

type EntryBuilder struct {
	// contains filtered or unexported fields
}

EntryBuilder provides methods to build ingestion control records.

func NewEntry 

func NewEntry(uniqifier int) *EntryBuilder

NewEntry starts building a new Entry.

func (*EntryBuilder) Build 

func (b *EntryBuilder) Build() *Entry

Build constructs the entry.

func (*EntryBuilder) WithBuildJoinedTime 

func (b *EntryBuilder) WithBuildJoinedTime(value time.Time) *EntryBuilder

WithBuildJoinedTime specifies the time the build result was populated.

func (*EntryBuilder) WithBuildProject 

func (b *EntryBuilder) WithBuildProject(project string) *EntryBuilder

WithBuildProject specifies the build project to use on the ingestion control record.

func (*EntryBuilder) WithBuildResult 

func (b *EntryBuilder) WithBuildResult(value *controlpb.BuildResult) *EntryBuilder

WithBuildResult specifies the build result for the entry.

func (*EntryBuilder) WithHasBuildbucketBuild 

func (b *EntryBuilder) WithHasBuildbucketBuild(hasBuildBucketBuild bool) *EntryBuilder

WithHasBuildbucketBuild specifies whether the build that is the subject of the ingestion has a buildbucket build.

func (*EntryBuilder) WithHasInvocation 

func (b *EntryBuilder) WithHasInvocation(hasInvocation bool) *EntryBuilder

WithHasInvocation specifies whether the build that is the subject of the ingestion has a ResultDB invocation.

func (*EntryBuilder) WithIngestionID 

func (b *EntryBuilder) WithIngestionID(id IngestionID) *EntryBuilder

WithIngestionID specifies the ingestion ID to use on the ingestion control record.

func (*EntryBuilder) WithInvocationJoinedTime 

func (b *EntryBuilder) WithInvocationJoinedTime(value time.Time) *EntryBuilder

WithInvocationJoinedTime specifies the time the invocation result was populated.

func (*EntryBuilder) WithInvocationProject 

func (b *EntryBuilder) WithInvocationProject(project string) *EntryBuilder

WithInvocationProject specifies the invocation project to use on the ingestion control record.

func (*EntryBuilder) WithInvocationResult 

func (b *EntryBuilder) WithInvocationResult(value *controlpb.InvocationResult) *EntryBuilder

WithInvocationResult specifies the invocation result for the entry.

func (*EntryBuilder) WithIsPresubmit 

func (b *EntryBuilder) WithIsPresubmit(isPresubmit bool) *EntryBuilder

WithIsPresubmit specifies whether the ingestion relates to a presubmit run.

func (*EntryBuilder) WithPresubmitJoinedTime 

func (b *EntryBuilder) WithPresubmitJoinedTime(lastUpdated time.Time) *EntryBuilder

WithPresubmitJoinedTime specifies the time the presubmit result was populated.

func (*EntryBuilder) WithPresubmitProject 

func (b *EntryBuilder) WithPresubmitProject(project string) *EntryBuilder

WithPresubmitProject specifies the presubmit project to use on the ingestion control record.

func (*EntryBuilder) WithPresubmitResult 

func (b *EntryBuilder) WithPresubmitResult(value *controlpb.PresubmitResult) *EntryBuilder

WithPresubmitResult specifies the build result for the entry.

type IngestionID 

type IngestionID string

IngestionID represents the join logic between invocation, build and cv runs. IngestionID can be derived from build ID or invocation ID. Entities with the same ingestionID belongs to the same ingestion.

func IngestionIDFromBuildID 

func IngestionIDFromBuildID(rdbHost string, buildID int64) IngestionID

IngestionIDFromBuildID returns the ingestionID for a given build id.

func IngestionIDFromInvocationID 

func IngestionIDFromInvocationID(rdbHost, invocationID string) IngestionID

IngestionIDFromInvocationID returns the ingestionID for a given invocation id.

type JoinStatistics 

type JoinStatistics struct {
	// TotalByHour captures the number of builds in the ingestionJoins
	// table eligible to be joined (i.e. have the left-hand join input).
	//
	// Data is broken down by by hours since the build became
	// eligible for joining. Index 0 indicates the period
	// from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on.
	TotalByHour []int64

	// JoinedByHour captures the number of builds in the ingestionJoins
	// table eligible to be joined, which were successfully joined (have
	// results for both join inputs present).
	//
	// Data is broken down by by hours since the build became
	// eligible for joining. Index 0 indicates the period
	// from ]-1 hour, now], index 1 indicates [-2 hour, -1 hour] and so on.
	JoinedByHour []int64
}

JoinStatistics captures indicators of how well two join inputs (e.g. buildbucket build completions and presubmit run completions, or buildbucket build completions and invocation finalizations) are being joined.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.