Documentation
¶
Overview ¶
Package common contains widely used CV utilities & types.
Index ¶
- Constants
- func CLIDsAsInt64s(ids []CLID) []int64
- func DifferenceSorted(a, b []int64) []int64
- func DistributeOffset(pollInterval time.Duration, keyParts ...string) time.Duration
- func IsDatastoreContention(err error) bool
- func IsDev(ctx context.Context) bool
- func LogError(ctx context.Context, err error, expectedErrors ...error)
- func MostSevereError(err error) error
- func SetDev(ctx context.Context) context.Context
- func TQifyError(ctx context.Context, err error) error
- func UnionSorted(a, b []int64) []int64
- func UniqueSorted(v []int64) []int64
- type CLID
- type CLIDs
- type RunID
- type RunIDs
- func (ids RunIDs) ContainsSorted(id RunID) bool
- func (p *RunIDs) DelSorted(id RunID) bool
- func (a RunIDs) DifferenceSorted(b RunIDs) RunIDs
- func (ids RunIDs) Equal(other RunIDs) bool
- func (ids RunIDs) Index(target RunID) int
- func (p *RunIDs) InsertSorted(id RunID)
- func (ids RunIDs) Len() int
- func (ids RunIDs) Less(i, j int) bool
- func (ids RunIDs) Set() map[RunID]struct{}
- func (ids RunIDs) Swap(i, j int)
- func (ids RunIDs) WithoutSorted(exclude RunIDs) RunIDs
- type TQIfy
Constants ¶
const ( // MaxTriggerAge limits CV operation only to most recently triggered CLs. // // In particular, this reduces Gerrit querying to CLs modified since // (now-MaxTriggerAge). MaxTriggerAge = 24 * 7 * time.Hour )
Variables ¶
This section is empty.
Functions ¶
func CLIDsAsInt64s ¶
CLIDsAsInt64s returns proto representation of CLIDs.
func DifferenceSorted ¶
DifferenceSorted returns all int64s in the first slice and not the second.
Both slices must be sorted. Doesn't modify input slices.
func DistributeOffset ¶
DistributeOffset deterministically chooses an offset across keys in [1..pollInterval) range aimining for uniform distribution across all keys.
A key is simply a concatenation of key parts with '\0' filler in between.
func IsDatastoreContention ¶
IsDatastoreContention is best-effort detection of transactions aborted due to pessimistic concurrency control of Datastore backed by Firestore.
This is fragile, because it relies on undocumented but likely rarely changed English description of an error.
func LogError ¶
LogError is errors.Log with CV-specific package filtering.
Logs entire error stack with ERROR severity by default. Logs just error with WARNING severity iff one of error (or its inner error) equal at least one of the given list of `expectedErrors` errors. This is useful if TQ handler is known to frequently fail this way.
expectedErrors must contain only unwrapped errors.
func MostSevereError ¶
MostSevereError returns the most severe error in order of non-transient => transient => nil.
Walks over potentially recursive errors.MultiError errors only.
Returns only singular errors or nil if input was nil.
func TQifyError ¶
TQifyError is shortcut for TQIfy{}.Error.
func UnionSorted ¶
UnionSorted returns sorted unique int64s from two slices.
Both slices must be sorted and unique. Doesn't modify input slices.
func UniqueSorted ¶
UniqueSorted sorts & removes duplicates in place.
Returns the potentially shorter slice.
Types ¶
type CLID ¶
type CLID int64
CLID is a unique ID of a CL used internally in CV.
It's just 8 bytes long and is thus much shorter than ExternalID, which reduces CPU & RAM & storage costs of CL graphs for multi-CL Runs.
type CLIDs ¶
type CLIDs []CLID
CLIDs is a convenience type to facilitate handling of a slice of CLID.
func (*CLIDs) Dedupe ¶
func (p *CLIDs) Dedupe()
Dedupe removes duplicates in place and sorts the slice.
Note: Does not preserve original order.
type RunID ¶
type RunID string
RunID is an unique RunID to identify a Run in CV.
RunID is string like `luciProject/timeComponent-1-hexHashDigest` consisting of 7 parts:
- The LUCI Project that this Run belongs to. Purpose: separates load on Datastore from different projects.
- `/` separator.
- (`endOfTheWorld` - CreateTime) in ms precision, left-padded with zeros to 13 digits. See `Run.CreateTime` Doc. Purpose: ensures queries by default orders runs of the same project by most recent first.
- `-` separator.
- Digest version (see part 7).
- `-` separator.
- A hex digest string uniquely identifying the set of CLs involved in this Run. Purpose: ensures two simultaneously started Runs in the same project won't have the same RunID.
func (RunID) AttemptKey ¶
AttemptKey returns CQDaemon attempt key.
type RunIDs ¶
type RunIDs []RunID
RunIDs is a convenience type to facilitate handling of run RunIDs.
func MakeRunIDs ¶
MakeRunIDs returns RunIDs from list of strings.
func (RunIDs) ContainsSorted ¶
ContainsSorted returns true if ids contain the given one.
func (*RunIDs) DelSorted ¶
DelSorted removes the given ID if it exists.
DelSorted is a pointer receiver method, because it modifies slice itself.
func (RunIDs) DifferenceSorted ¶
DifferenceSorted returns all IDs in this slice and not the other one.
Both slices must be sorted. Doesn't modify input slices.
func (RunIDs) Index ¶
Index returns the index of the first instance of the provided id.
Returns -1 if the provided id isn't present.
func (*RunIDs) InsertSorted ¶
InsertSorted adds given ID if not yet exists to the list keeping list sorted.
InsertSorted is a pointer receiver method, because it modifies slice itself.
func (RunIDs) WithoutSorted ¶
WithoutSorted returns a subsequence of IDs without excluded IDs.
Both this and the excluded slices must be sorted.
If this and excluded IDs are disjoint, return this slice. Otherwise, returns a copy without excluded IDs.
type TQIfy ¶
type TQIfy struct {
// KnownRetry are expected errors which will result in HTTP 429 and retries.
//
// Retries may not happen if task queue configuration prevents it, e.g.
// because task has exhausted its retry quota.
//
// KnownRetry and KnownIgnore should not match the same error, but if this
// happens, Retry takes effect and KnownIgnore is ignored to avoid accidental
// loss of tasks.
//
// Must contain only leaf errors, i.e. no annotated or MultiError objects.
KnownRetry []error
// KnownRetryTags are similar to `KnowRetry`, but are the expected tags that
// the CV error should be tagged with.
//
// Must not contain `transient.Tag`.
KnownRetryTags []errors.BoolTag
// NeverRetry instructs TQ not to retry on any unexpected error.
//
// Transient error will be tagged with `tq.Ignore` while non-transient error
// will be tagged with `tq.Fatal`. See the struct doc for what each tag means.
//
// Recommend to use this flag when tasks are executed periodically in short
// interval (e.g. refresh config task) where as retrying failed task is not
// necessary.
//
// Mutually exclusive with `KnownRetry` and `KnownRetryTags`.
NeverRetry bool
// KnownIgnore are expected errors which will result in HTTP 204 and no
// retries.
//
// Must contain only leaf errors, i.e. no annotated or MultiError objects.
KnownIgnore []error
// KnownIgnoreTags are similar to `KnownIgnore`, but are the expected tags
// that the CV error should be tagged with.
//
// Must not contain `transient.Tag`.
KnownIgnoreTags []errors.BoolTag
}
TQIfy converts CV error semantics to server/TQ, and logs error if necessary.
Usage:
func tqHandler(ctx ..., payload...) error {
err := doStuff(ctx, ...)
return TQIfy{}.Error(ctx, err)
}
Given that:
- TQ lib recognizes these error kinds:
- tq.Ignore => HTTP 204, no retries
- tq.Fatal => HTTP 202, no retries, but treated with alertable in our monitoring configuration;
- transient.Tag => HTTP 500, will be retried;
- else => HTTP 429, will be retried.
OTOH, CV uses
- transient.Tag to treat all _transient_ situations, where retry should help
- else => permanent errors, where retries aren't helpful.
Most _transient_ situations in CV are due to expected issues such as Gerrit giving stale data. Getting HTTP 500s in this case is an unfortunate noise, which obscures other infrequent situations which are worth looking at.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bq handles sending rows to BigQuery.
|
Package bq handles sending rows to BigQuery. |
|
Package eventbox batches incoming events for a single Datastore entity for processing.
|
Package eventbox batches incoming events for a single Datastore entity for processing. |
|
dsset
Package dsset implements a particular flavor of Datastore-on-Firestore backed set.
|
Package dsset implements a particular flavor of Datastore-on-Firestore backed set. |
|
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource.
|
Package lease provides a way to "lock" an external resource with expiration time so that concurrent processes/task executions can achieve exclusive privilege to make mutations (generally long-running and non-idempotent) on that resource. |
|
Package tree implements fetching tree status from Tree Status App.
|
Package tree implements fetching tree status from Tree Status App. |
|
treetest
Package treetest implements fake Tree for testing in CV.
|
Package treetest implements fake Tree for testing in CV. |