common

package
v0.0.0-...-add0f8d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2024 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package common contains widely used CV utilities & types.

Index

Constants

View Source 
const (
	// MaxTriggerAge limits CV operation only to most recently triggered CLs.
	//
	// In particular, this reduces Gerrit querying to CLs modified since
	// (now-MaxTriggerAge).
	MaxTriggerAge = 24 * 7 * time.Hour

	// LaunchRetryClockTag tags the clock to coordinate with the test timer
	// call back in cvtesting.
	LaunchRetryClockTag = "retry-launch"
)
View Source 
const (
	// FooterNoEquivalentBuilders instructs LUCI CV to ignore the equivalent
	// builder. Meaning always launch the main builder if possible.
	FooterNoEquivalentBuilders = "No-Equivalent-Builders"
	// FooterLegacyNoEquivalentBuilders is the legacy version of
	// FooterNoEquivalentBuilders
	FooterLegacyNoEquivalentBuilders = "NO_EQUIVALENT_BUILDERS"
	// FooterCQDoNotCancelTryjobs instructs LUCI CV to not to cancel the Tryjobs.
	FooterCQDoNotCancelTryjobs = "Cq-Do-Not-Cancel-Tryjobs"
	// FooterNoTreeChecks instructs LUCI CV to skip tree check before submission.
	FooterNoTreeChecks = "No-Tree-Checks"
	// FooterLegacyNoTreeChecks is the legacy version of FooterNoTreeChecks.
	FooterLegacyNoTreeChecks = "NOTREECHECKS"
	// FooterNoTry instructs LUCI CV to skip all Tryjobs except Presubmit.
	FooterNoTry = "No-Try"
	// FooterLegacyNoTry is the legacy version of FooterNoTry
	FooterLegacyNoTry = "NOTRY"
	// FooterNoPresubmit instructs LUCI CV to skip the Presubmit Tryjob.
	//
	// CAVEAT: crbug.com/1292195 - It use `disable_reuse` field to decide
	// whether a Tryjob is Presubmit which produce false positive.
	FooterNoPresubmit = "No-Presubmit"
	// FooterLegacyPresubmit is the legacy version of FooterNoPresubmit.
	FooterLegacyPresubmit = "NOPRESUBMIT"
	// FooterCQIncludeTryjobs specifies the additional Tryjobs to launch.
	FooterCQIncludeTryjobs = "Cq-Include-Trybots"
	// FooterLegacyCQIncludeTryjobs is the legacy version of
	// FooterCQIncludeTryjobs.
	FooterLegacyCQIncludeTryjobs = "CQ_INCLUDE_TRYBOTS"
	// FooterOverrideTryjobsForAutomation provides an list of Tryjobs that
	// overrides ALL the Tryjobs supposed to be launched according to the
	// configuration.
	FooterOverrideTryjobsForAutomation = "Override-Tryjobs-For-Automation"
	// FooterCQClTag specifies the additional Tag that will be added to the
	// launched Tryjobs.
	FooterCQClTag = "Cq-Cl-Tag"
)
View Source 
const InstantTriggerDogfooderGroup = "luci-cv-instant-trigger-dogfooders"

InstantTriggerDogfooderGroup is the CrIA group who signed up for dogfooding cros instant trigger.

View Source 
const MaxRunTotalDuration = 10 * 24 * time.Hour // 10 days

MaxRunTotalDuration is the max total duration of the Run.

Total duration means end time - create time. Run will be cancelled after the total duration is reached.

View Source 
const RunKind = "Run"

RunKind is the Datastore entity kind for Run.

Variables

View Source 
var DSContentionTag = errors.BoolTag{Key: errors.NewTagKey("Datastore Contention")}

DSContentionTag when set indicates Datastore contention.

It's set on errors by parts of CV which are especially prone to DS contention to reduce noise in logs and for more effective retries.

Functions

func CLIDsAsInt64s 

func CLIDsAsInt64s(ids []CLID) []int64

CLIDsAsInt64s returns proto representation of CLIDs.

func DifferenceSorted 

func DifferenceSorted(a, b []int64) []int64

DifferenceSorted returns all int64s in the first slice and not the second.

Both slices must be sorted. Doesn't modify input slices.

func DistributeOffset 

func DistributeOffset(pollInterval time.Duration, keyParts ...string) time.Duration

DistributeOffset deterministically chooses an offset across keys in [1..pollInterval) range aimining for uniform distribution across all keys.

A key is simply a concatenation of key parts with '\0' filler in between.

func IsDatastoreContention 

func IsDatastoreContention(err error) bool

IsDatastoreContention is best-effort detection of transactions aborted due to pessimistic concurrency control of Datastore backed by Firestore.

This is fragile, because it relies on undocumented but likely rarely changed English description of an error.

func IsInstantTriggerDogfooder 

func IsInstantTriggerDogfooder(ctx context.Context, id identity.Identity) bool

IsInstantTriggerDogfooder returns true if the given user participate in the cros instant trigger dogfood.

TODO(yiwzhang): remove this function, once cros instant trigger dogfood is done.

func LogError 

func LogError(ctx context.Context, err error, expectedErrors ...error)

LogError is errors.Log with CV-specific package filtering.

Logs entire error stack with ERROR severity by default. Logs just error with WARNING severity iff one of error (or its inner error) equal at least one of the given list of `expectedErrors` errors. This is useful if TQ handler is known to frequently fail this way.

expectedErrors must contain only unwrapped errors.

func MostSevereError 

func MostSevereError(err error) error

MostSevereError returns the most severe error in order of non-transient => transient => nil.

Walks over potentially recursive errors.MultiError errors only.

Returns only singular errors or nil if input was nil.

func PB2TimeNillable 

func PB2TimeNillable(pb *timestamppb.Timestamp) time.Time

PB2TimeNillable is the opposite of Time2PBNillable.

ie same as pb.AsTime() but returns zero time if pb is nil.

func TQifyError 

func TQifyError(ctx context.Context, err error) error

TQifyError is shortcut for TQIfy{}.Error.

func Time2PBNillable 

func Time2PBNillable(t time.Time) *timestamppb.Timestamp

Time2PBNillable is like timestamppb.New() but returns nil on zero time.

func UnionSorted 

func UnionSorted(a, b []int64) []int64

UnionSorted returns sorted unique int64s from two slices.

Both slices must be sorted and unique. Doesn't modify input slices.

func UniqueSorted 

func UniqueSorted(v []int64) []int64

UniqueSorted sorts & removes duplicates in place.

Returns the potentially shorter slice.

Types

type CLID 

type CLID int64

CLID is a unique ID of a CL used internally in CV.

It's just 8 bytes long and is thus much shorter than ExternalID, which reduces CPU & RAM & storage costs of CL graphs for multi-CL Runs.

type CLIDs 

type CLIDs []CLID

CLIDs is a convenience type to facilitate handling of a slice of CLID.

func MakeCLIDs 

func MakeCLIDs(ids ...int64) CLIDs

MakeCLIDs returns CLIDs from list of clids in int64.

func (CLIDs) Contains 

func (ids CLIDs) Contains(id CLID) bool

Contains returns true if CLID is inside these CLIDs.

func (*CLIDs) Dedupe 

func (p *CLIDs) Dedupe()

Dedupe removes duplicates in place and sorts the slice.

Note: Does not preserve original order.

func (CLIDs) Len 

func (ids CLIDs) Len() int

Len is the number of elements in the collection.

func (CLIDs) Less 

func (ids CLIDs) Less(i int, j int) bool

Less reports whether the element with index i should sort before the element with index j.

func (CLIDs) Set 

func (ids CLIDs) Set() CLIDsSet

Set returns a new set of CLIDs.

func (CLIDs) Swap 

func (ids CLIDs) Swap(i int, j int)

Swap swaps the elements with indexes i and j.

type CLIDsSet 

type CLIDsSet map[CLID]struct{}

CLIDsSet is convenience type to reduce the boilerplate.

func MakeCLIDsSet 

func MakeCLIDsSet(ids ...int64) CLIDsSet

MakeCLIDsSet returns new CLIDsSet from list of clids in int64.

func (CLIDsSet) Add 

func (s CLIDsSet) Add(clid CLID)

func (CLIDsSet) AddI64 

func (s CLIDsSet) AddI64(id int64)

func (CLIDsSet) Del 

func (s CLIDsSet) Del(id CLID)

func (CLIDsSet) DelAll 

func (s CLIDsSet) DelAll(ids CLIDs)

func (CLIDsSet) DelI64 

func (s CLIDsSet) DelI64(id int64)

func (CLIDsSet) Has 

func (s CLIDsSet) Has(clid CLID) bool

func (CLIDsSet) HasI64 

func (s CLIDsSet) HasI64(id int64) bool

func (CLIDsSet) Reset 

func (s CLIDsSet) Reset(ids ...CLID)

Reset resets the set to contain just the given IDs.

func (CLIDsSet) ResetI64 

func (s CLIDsSet) ResetI64(ids ...int64)

func (CLIDsSet) ToCLIDs 

func (s CLIDsSet) ToCLIDs() CLIDs

type Env 

type Env struct {
	// LogicalHostname is CV hostname referred to in configs.
	//
	// On GAE, this is something like "luci-change-verifier-dev.appspot.com"
	// and it is part of the HTTPAddressBase.
	//
	// Under local development, this is usually set to GAE-looking hostname, while
	// keeping HTTPAddressBase a real localhost URL.
	LogicalHostname string

	// HTTPAddressBase can be used to generate URLs to this CV service.
	//
	// Doesn't have a trailing slash.
	//
	// For example,
	//   * "https://luci-change-verifier-dev.appspot.com"
	//   * "http://localhost:8080"
	HTTPAddressBase string

	// IsGAEDev is true if this is a -dev GAE environment.
	//
	// Deprecated. Do not use in new code. It should only be used during migration
	// from CQDaemon which doesn't have equivalent -dev environment.
	IsGAEDev bool

	// GAEInfo is populated if LUCI CV runs on GAE.
	GAEInfo struct {
		// CloudProject is the name of the Google Cloud Project LUCI CV runs in.
		CloudProject string
		// ServiceName is the name of the micro-service in the GAE app.
		ServiceName string
		// InstanceID is the ID of the instance that runs LUCI CV.
		InstanceID string
	}
}

Env describes where CV runs at.

func MakeEnv 

func MakeEnv(opts server.Options) *Env

MakeEnv creates a new `Env` from server options.

type RunID 

type RunID string

RunID is an unique RunID to identify a Run in CV.

RunID is string like `luciProject/inverseTS-1-hexHashDigest` consisting of 7 parts:

  1. The LUCI Project that this Run belongs to. Purpose: separates load on Datastore from different projects.
  2. `/` separator.
  3. InverseTS, defined as (`endOfTheWorld` - CreateTime) in ms precision, left-padded with zeros to 13 digits. See `Run.CreateTime` Doc. Purpose: ensures queries by default orders runs of the same project by most recent first.
  4. `-` separator.
  5. Digest version (see part 7).
  6. `-` separator.
  7. A hex digest string uniquely identifying the set of CLs involved in this Run. Purpose: ensures two simultaneously started Runs in the same project won't have the same RunID.

func FromPublicRunID 

func FromPublicRunID(id string) (RunID, error)

FromPublicRunID is the inverse of RunID.PublicID().

func MakeRunID 

func MakeRunID(luciProject string, createTime time.Time, digestVersion int, clsDigest []byte) RunID

func (RunID) AttemptKey 

func (id RunID) AttemptKey() string

AttemptKey returns CQDaemon attempt key.

func (RunID) Inner 

func (id RunID) Inner() string

Inner is the part after "<LUCIProject>/" for use in UI.

func (RunID) InverseTS 

func (id RunID) InverseTS() string

InverseTS of this Run. See RunID doc.

func (RunID) LUCIProject 

func (id RunID) LUCIProject() string

LUCIProject this Run belongs to.

func (RunID) PublicID 

func (id RunID) PublicID() string

PublicID returns the public representation of the RunID.

The format of a public ID is `projects/$luci-project/runs/$id`, where - luci-project is the name of the LUCI project the Run belongs to - id is an opaque key unique in the LUCI project.

func (RunID) Validate 

func (id RunID) Validate() (err error)

Validate returns an error if Run ID is not valid.

If validate returns nil,

  • it means all other methods on RunID will work fine instead of panicking,
  • it doesn't mean Run ID is possible to generate using the MakeRunID. This is especially relevant in CV tests, where specifying short Run IDs is useful.

type RunIDs 

type RunIDs []RunID

RunIDs is a convenience type to facilitate handling of run RunIDs.

func MakeRunIDs 

func MakeRunIDs(ids ...string) RunIDs

MakeRunIDs returns RunIDs from list of strings.

func (RunIDs) ContainsSorted 

func (ids RunIDs) ContainsSorted(id RunID) bool

ContainsSorted returns true if ids contain the given one.

func (*RunIDs) DelSorted 

func (p *RunIDs) DelSorted(id RunID) bool

DelSorted removes the given ID if it exists.

DelSorted is a pointer receiver method, because it modifies slice itself.

func (RunIDs) DifferenceSorted 

func (a RunIDs) DifferenceSorted(b RunIDs) RunIDs

DifferenceSorted returns all IDs in this slice and not the other one.

Both slices must be sorted. Doesn't modify input slices.

func (RunIDs) Equal 

func (ids RunIDs) Equal(other RunIDs) bool

Equal checks if two ids are equal.

func (RunIDs) Index 

func (ids RunIDs) Index(target RunID) int

Index returns the index of the first instance of the provided id.

Returns -1 if the provided id isn't present.

func (*RunIDs) InsertSorted 

func (p *RunIDs) InsertSorted(id RunID)

InsertSorted adds given ID if not yet exists to the list keeping list sorted.

InsertSorted is a pointer receiver method, because it modifies slice itself.

func (RunIDs) Len 

func (ids RunIDs) Len() int

func (RunIDs) Less 

func (ids RunIDs) Less(i, j int) bool

sort.Interface copy-pasta.

func (RunIDs) Set 

func (ids RunIDs) Set() map[RunID]struct{}

Set returns a new set of run IDs.

func (RunIDs) Swap 

func (ids RunIDs) Swap(i, j int)

func (RunIDs) WithoutSorted 

func (ids RunIDs) WithoutSorted(exclude RunIDs) RunIDs

WithoutSorted returns a subsequence of IDs without excluded IDs.

Both this and the excluded slices must be sorted.

If this and excluded IDs are disjoint, return this slice. Otherwise, returns a copy without excluded IDs.

type TQIfy 

type TQIfy struct {
	// KnownRetry are expected errors which will result in HTTP 429 and retries.
	//
	// Retries may not happen if task queue configuration prevents it, e.g.
	// because task has exhausted its retry quota.
	//
	// KnownRetry and KnownIgnore should not match the same error, but if this
	// happens, Retry takes effect and KnownIgnore is ignored to avoid accidental
	// loss of tasks.
	//
	// Must contain only leaf errors, i.e. no annotated or MultiError objects.
	KnownRetry []error
	// KnownRetryTags are similar to `KnowRetry`, but are the expected tags that
	// the CV error should be tagged with.
	//
	// Must not contain `transient.Tag`.
	KnownRetryTags []errors.BoolTag
	// NeverRetry instructs TQ not to retry on any unexpected error.
	//
	// Transient error will be tagged with `tq.Ignore` while non-transient error
	// will be tagged with `tq.Fatal`. See the struct doc for what each tag means.
	//
	// Recommend to use this flag when tasks are executed periodically in short
	// interval (e.g. refresh config task) where as retrying failed task is not
	// necessary.
	//
	// Mutually exclusive with `KnownRetry` and `KnownRetryTags`.
	NeverRetry bool
	// KnownIgnore are expected errors which will result in HTTP 204 and no
	// retries.
	//
	// Must contain only leaf errors, i.e. no annotated or MultiError objects.
	KnownIgnore []error
	// KnownIgnoreTags are similar to `KnownIgnore`, but are the expected tags
	// that the CV error should be tagged with.
	//
	// Must not contain `transient.Tag`.
	KnownIgnoreTags []errors.BoolTag
}

TQIfy converts CV error semantics to server/TQ, and logs error if necessary.

Usage: 

func tqHandler(ctx ..., payload...) error {
  err := doStuff(ctx, ...)
  return TQIfy{}.Error(ctx, err)
}

Given that:

  • TQ lib recognizes these error kinds:
  • tq.Ignore => HTTP 204, no retries
  • tq.Fatal => HTTP 202, no retries, but treated with alertable in our monitoring configuration;
  • transient.Tag => HTTP 500, will be retried;
  • else => HTTP 429, will be retried.

OTOH, CV uses

  • transient.Tag to treat all _transient_ situations, where retry should help
  • else => permanent errors, where retries aren't helpful.

Most _transient_ situations in CV are due to expected issues such as Gerrit giving stale data. Getting HTTP 500s in this case is an unfortunate noise, which obscures other infrequent situations which are worth looking at.

func (TQIfy) Error 

func (t TQIfy) Error(ctx context.Context, err error) error

type TryjobID 

type TryjobID int64

TryjobID is a unique ID of a Tryjob used internally in CV.

This ID is not a Buildbucket Build ID. See also tryjob.Tryjob type.

type TryjobIDSet 

type TryjobIDSet map[TryjobID]struct{}

TryjobIDSet is convenience type to reduce the boilerplate.

func (TryjobIDSet) Add 

func (s TryjobIDSet) Add(tjID TryjobID)

Add adds the provided Tryjob ID to the set.

func (TryjobIDSet) Has 

func (s TryjobIDSet) Has(tjID TryjobID) bool

Has returns true if the provided Tryjob ID is in the set.

Otherwise, returns false.

type TryjobIDs 

type TryjobIDs []TryjobID

TryjobIDs is a convenience type to facilitate handling of a slice of TryjobID.

func MakeTryjobIDs 

func MakeTryjobIDs(ids ...int64) TryjobIDs

MakeTryjobIDs returns TryjobIDs from list of TryjobID in int64.

func (*TryjobIDs) Dedupe 

func (p *TryjobIDs) Dedupe()

Dedupe removes duplicates in place and sorts the slice.

Note: Does not preserve original order.

func (TryjobIDs) ToInt64 

func (ids TryjobIDs) ToInt64() []int64

ToInt64 returns a slice that contains all Tryjobs in int64 type.

Source Files

View all Source files

Directories

Path Synopsis
Package bq handles sending rows to BigQuery.
 Package bq handles sending rows to BigQuery.
Package eventbox batches incoming events for a single Datastore entity for processing.
 Package eventbox batches incoming events for a single Datastore entity for processing.
dsset
Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
 Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
 Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
Package pubsub provides a generic way to batch pubsub pull notifications.
 Package pubsub provides a generic way to batch pubsub pull notifications.
Package tree implements fetching tree status from Tree Status App.
 Package tree implements fetching tree status from Tree Status App.
treetest
Package treetest implements fake Tree for testing in CV.
 Package treetest implements fake Tree for testing in CV.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.