model

package
v0.0.0-...-a70aae3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 14, 2024 License: Apache-2.0 Imports: 37 Imported by: 0

Documentation

Overview

Package model contains definition of Swarming Datastore entities.

The source of truth (with all detailed comments) is still in luci-py. Here only entities that the Go side reads are duplicated. Many fields are omitted.

All writes happen through Python side.

Index

Constants

View Source 
const (
	// ChunkSize is the size of each TaskOutputChunk.Chunk in uncompressed form.
	//
	// The rationale is that appending data to an entity requires reading it
	// first, so it must not be too big. On the other hand, having thousands of
	// small entities is pure overhead.
	//
	// Note that changing this value is a breaking change for existing logs.
	ChunkSize = 100 * 1024

	// MaxChunks sets a limit on how much of a task output to store.
	MaxChunks = 1024
)
View Source 
const (
	// TaskToRunShards is the number of TaskToRun entity kinds to shard across.
	TaskToRunShards = 16
)

Variables

View Source 
var (
	// BeginningOfTheWorld is used as beginning of time when constructing request
	// keys: number of milliseconds since BeginningOfTheWorld is part of the key.
	//
	// The world started on 2010-01-01 at 00:00:00 UTC. The rationale is that
	// using the original Unix epoch (1970) results in 40 years worth of key space
	// wasted.
	//
	// We allocate 43 bits in the key for storing the timestamp at millisecond
	// precision. This makes this scheme good for 2**43 / 365 / 24 / 3600 / 1000,
	// which 278 years. We'll have until 2010+278 = 2288 before we run out of
	// key space. Should be good enough for now. Can be fixed later.
	BeginningOfTheWorld = time.Date(2010, 1, 1, 0, 0, 0, 0, time.UTC)
)
View Source 
var ErrNoSuchFetchOperation = errors.New("no such fetch operation")

ErrNoSuchFetchOperation is returned when trying to get result of an unknown fetch operation.

Functions

func BotEventsQuery 

func BotEventsQuery(ctx context.Context, botID string) *datastore.Query

BotEventsQuery prepares a query that fetches BotEvent entities for a bot.

Most recent events are returned first.

func BotInfoKey 

func BotInfoKey(ctx context.Context, botID string) *datastore.Key

BotInfoKey builds a BotInfo key given the bot ID.

func BotInfoQuery 

func BotInfoQuery() *datastore.Query

BotInfoQuery prepares a query that fetches BotInfo entities.

func BotRootKey 

func BotRootKey(ctx context.Context, botID string) *datastore.Key

BotRootKey is a root key of an entity group with info about a bot.

func BotsDimensionsAggregationInfoKey 

func BotsDimensionsAggregationInfoKey(ctx context.Context) *datastore.Key

BotsDimensionsAggregationInfoKey is BotsDimensionsAggregationInfo entity key.

func BotsDimensionsAggregationKey 

func BotsDimensionsAggregationKey(ctx context.Context) *datastore.Key

BotsDimensionsAggregationKey is BotsDimensionsAggregation entity key.

func BuildTaskKey 

func BuildTaskKey(ctx context.Context, taskReq *datastore.Key) *datastore.Key

BuildTaskKey construct a BuildTask key given a task request key.

func DimensionsFlatToPb 

func DimensionsFlatToPb(flat []string) []*apipb.StringListPair

DimensionsFlatToPb converts a list of k:v pairs into []*apipb.StringListPair.

func FilterBotsByDimensions 

func FilterBotsByDimensions(q *datastore.Query, mode SplitMode, dims Filter) []*datastore.Query

FilterBotsByDimensions limits a BotInfo query to return bots matching these dimensions.

For complex filters this may split the query into multiple queries that need to run in parallel with their results merged. See SplitForQuery() in Filter for more details.

func FilterBotsByState 

func FilterBotsByState(q *datastore.Query, state StateFilter) *datastore.Query

FilterBotsByState limits a BotInfo query to return bots in particular state.

func FilterTasksByCreationTime 

func FilterTasksByCreationTime(ctx context.Context, q *datastore.Query, start, end time.Time, cursor *cursorpb.TasksCursor) (*datastore.Query, error)

FilterTasksByCreationTime limits a TaskResultSummary or TaskRunResult query to return tasks created within the given time range [start, end), applying a task pagination cursor as well if given (since it also just modifies the query time range).

Works only for queries ordered by key. Returns an error if timestamps are outside of expected range (e.g. before BeginningOfTheWorld).

Timestamp precision is truncated to milliseconds. Zero time means no limit on the corresponding side of the range.

Returns datastore.ErrNullQuery if the pagination cursor points exactly at the `start` end of the query's time range. This is rare, but possible. Such query will always produce no results (we've "seen" the last result already, since it is in the cursor) and can be skipped.

func FilterTasksByTags 

func FilterTasksByTags(q *datastore.Query, mode SplitMode, tags Filter) []*datastore.Query

FilterTasksByTags limits a TaskResultSummary query to return tasks matching given tags filter.

For complex filters this may split the query into multiple queries that need to run in parallel with their results merged. See SplitForQuery() in Filter for more details.

func FilterTasksByTimestampField 

func FilterTasksByTimestampField(q *datastore.Query, field string, start, end time.Time) *datastore.Query

FilterTasksByTimestampField orders the query by a timestamp stored in the given field, filtering based on it as well.

Tasks will be returned in the descending order (i.e. the most recent first).

The filter range is [start, end). Zero time means no limit on the corresponding side of the range.

func FromJSONProperty 

func FromJSONProperty(prop datastore.Property, val any) error

FromJSONProperty deserializes a JSON blob property into `val`.

If the property is missing, `val` will be unchanged. Assumes `val` is a list or a dict.

Recognizes zlib-compressed properties for compatibility with older entities.

func GenerateBootstrapToken 

func GenerateBootstrapToken(ctx context.Context, caller identity.Identity) (string, error)

GenerateBootstrapToken generates a token used to authenticate bot code fetch requests used during the bot bootstrap.

func LegacyBootstrapSecretKey 

func LegacyBootstrapSecretKey(ctx context.Context) *datastore.Key

LegacyBootstrapSecretKey is a datastore key of the bootstrap secret.

func MapToStringListPair 

func MapToStringListPair(p map[string][]string, keySorting bool) []*apipb.StringListPair

MapToStringListPair converts a map[string][]string to []*apipb.StringListPair. If keySorting, sorting is applied to the keys.

func NewTaskRequestID 

func NewTaskRequestID(ctx context.Context) int64

NewTaskRequestID generates an ID for a new task.

func PerformanceStatsKey 

func PerformanceStatsKey(ctx context.Context, taskReq *datastore.Key) *datastore.Key

PerformanceStatsKey builds a PerformanceStats key given a task request key.

func RequestKeyToTaskID 

func RequestKeyToTaskID(key *datastore.Key, variant TaskIDVariant) string

RequestKeyToTaskID converts TaskRequest entity key to a string form used in external APIs.

For legacy reasons they are two flavors of string task IDs:

  1. A "packed TaskRequest key", aka "packed TaskResultSummary" key. It is a hex string ending with 0, e.g. `6663cfc78b41fb10`. Pass AsRequest as the second argument to request this variant.
  2. A "packed TaskRunResult key". It is a hex string ending with 1, e.g. `6663cfc78b41fb11`. Pass AsRunResult as the second argument to request this variant.

Some APIs return the first form, others return the second. There's no clear logical reason why they do so anymore. They do it for backward compatibility with much older API, where these differences mattered.

Panics if `key` is not a TaskRequest key.

func SecretBytesKey 

func SecretBytesKey(ctx context.Context, taskReq *datastore.Key) *datastore.Key

SecretBytesKey constructs SecretBytes key given a TaskRequest key.

func SetBBTaskStatus 

func SetBBTaskStatus(state apipb.TaskState, failure bool, bbTask *bbpb.Task)

SetBBTaskStatus converts a swarming task result's state to a buildbucket status, updates StatusDetails and SummaryMarkdown accordingly.

It Modifies the given *bbpb.Task in place.

func SortStringPairs 

func SortStringPairs(pairs []*apipb.StringPair)

SortStringPairs sorts string pairs. This was stolen from go.chromium.org/luci/buildbucket/protoutil/tag.go and should probably be moved to go.chromium.org/luci/common, but that would require a larger refactor, hence the following: TODO (crbug.com/1508908): remove this once refactored.

func SpecName 

func SpecName(tagsMap map[string]string) string

SpecName extracts the spec name from the tags map.

func TagListToMap 

func TagListToMap(tags []string) (tagsMap map[string]string)

TagListToMap converts tags in []string format to a string map.

func TaskIDToRequestKey 

func TaskIDToRequestKey(ctx context.Context, taskID string) (*datastore.Key, error)

TaskIDToRequestKey returns TaskRequest entity key given a task ID string.

The task ID is something that looks like `6663cfc78b41fb10`, it is either a "packed TaskRequest key" (when ends with 0) or "a packed TaskRunResult key" (when ends with non-0). See RequestKeyToTaskID.

Task request key is a root key of the hierarchy of entities representing a particular task. All key constructor functions for such entities take the request key as an argument.

func TaskOutputChunkKey 

func TaskOutputChunkKey(ctx context.Context, taskReq *datastore.Key, idx int64) *datastore.Key

TaskOutputChunkKey builds a TaskOutputChunk key for the given chunk index.

func TaskRequestIDKey 

func TaskRequestIDKey(ctx context.Context, requestID string) *datastore.Key

TaskRequestIDKey constructs a top-level TaskRequestID key.

func TaskRequestToToRunKey 

func TaskRequestToToRunKey(ctx context.Context, taskReq *TaskRequest, sliceIndex int) (*datastore.Key, error)

TaskRequestToToRunKey builds a TaskToRun key given the task request and the slice index.

func TaskResultSummaryKey 

func TaskResultSummaryKey(ctx context.Context, taskReq *datastore.Key) *datastore.Key

TaskResultSummaryKey construct a summary key given a task request key.

func TaskResultSummaryQuery 

func TaskResultSummaryQuery() *datastore.Query

TaskResultSummaryQuery prepares a query that fetches TaskResultSummary entities.

func TaskRunResultKey 

func TaskRunResultKey(ctx context.Context, taskReq *datastore.Key) *datastore.Key

TaskRunResultKey constructs a task run result key given a task request key.

This is purely a key constructor. It doesn't fetch anything and doesn't handle deduplicated tasks.

func TaskRunResultMissingFieldsFromRequest 

func TaskRunResultMissingFieldsFromRequest(pb *apipb.TaskResultResponse, req *TaskRequest)

TaskRunResultMissingFieldsFromRequest sets fields that TaskRunResult.ToProto could not set if the entity is too old.

func TaskRunResultQuery 

func TaskRunResultQuery() *datastore.Query

TaskRunResultQuery prepares a query that fetches TaskRunResult entities.

func TaskToRunKey 

func TaskToRunKey(ctx context.Context, taskReq *datastore.Key, shardIdx int32, ttrID int64) *datastore.Key

TaskToRunKey builds a TaskToRun key given the task request key, the entity kind shard index and the task to run ID.

func TaskToRunKind 

func TaskToRunKind(shardIdx int32) string

TaskToRunKind returns the TaskToRun entity kind name given a shard index.

func TimestampToRequestKey 

func TimestampToRequestKey(ctx context.Context, timestamp time.Time, suffix int64) (*datastore.Key, error)

TimestampToRequestKey converts a timestamp to a request key.

Note that this function does NOT accept a task id. This functions is primarily meant for limiting queries to a task creation time range.

func ToJSONProperty 

func ToJSONProperty(val any) (datastore.Property, error)

ToJSONProperty serializes a value into a JSON blob property.

Empty maps and lists are stored as nulls.

func ValidateBootstrapToken 

func ValidateBootstrapToken(ctx context.Context, tok string) (identity.Identity, error)

ValidateBootstrapToken checks if the bootstrap token is valid.

If the token is valid returns the identity of whoever generated it. Returns a fatal error if the token is invalid or a transient error if the validation process itself failed (e.g. a datastore error).

Types

type BatchFetcher 

type BatchFetcher[K comparable, E any] struct {
	// contains filtered or unexported fields
}

BatchFetcher can fetch entities by key, in batches.

This allows to relatively efficiently "join" results of some datastore query with another set of entities.

It is a map from some `K` (that is just a local key, can be anything at all as long as it is comparable) to an entity-to-be-fetched `*E`. The entity key is part of `E`.

The map is populated incrementally via Fetch(key, &Entity{...}) calls. Then Wait is called to wait for all pending fetches to complete. Then results are available via Get(key) calls.

The advantage of that approach over just making a single multi-get is that we can start fetching things earlier, while still iterating over the query results.

func NewBatchFetcher 

func NewBatchFetcher[K comparable, E any](ctx context.Context, batchSize, concurrencyLimit int) *BatchFetcher[K, E]

NewBatchFetcher creates a batch fetcher.

It should be used only for one fetch session. It uses the given context for all operations.

Close must be called at some point to shutdown internal goroutines. It is OK to call it multiple times or from a defer.

Recommended batchSize is 300 to match the default datastore query page size. In that case batches will be fetched essentially in parallel with query pages.

Concurrency limit puts a limit on a number of concurrently running fetches (to avoid consuming excessive resources).

func (*BatchFetcher[K, E]) Close 

func (b *BatchFetcher[K, E]) Close()

Close cancels all pending and running fetch operations (if any).

It is fine to call it multiple times.

func (*BatchFetcher[K, E]) Fetch 

func (b *BatchFetcher[K, E]) Fetch(key K, entity *E)

Fetch schedules fetching of the given entity.

It will be fetched as a part of a batch multi-get call when the number of pending entities is equal to the configured batch size. Blocks if there are already more than configured number of concurrent multi-get calls happening right now.

The result can be retrieved by Get(key), but only after Wait is called.

Panics if there's already a fetch operation with the given key. Panics if the fetcher is already stopped (i.e. Wait or Close was called).

func (*BatchFetcher[K, E]) Get 

func (b *BatchFetcher[K, E]) Get(key K) (*E, error)

Get returns the fetched entity for the given operation key.

Must be called only after Wait(). Panics otherwise.

Returns an error if this entity could not be fetched. In particular the error is ErrNoSuchEntity if there's no such entity. Returns a context error if the batcher was closed (or its underlying context has expired) before the entity could be fetched.

Returns ErrNoSuchFetchOperation if this operation key is unknown.

func (*BatchFetcher[K, E]) Wait 

func (b *BatchFetcher[K, E]) Wait()

Wait waits for all fetch operations to complete.

Should be called before Get can be used. It is fine to call it multiple times (redundant calls will just do nothing).

If there were any errors while fetching entities (including ErrNoSuchEntity errors or context cancellation errors), they will be returned by the corresponding Get() calls.

type BotCommon 

type BotCommon struct {
	// State is a free form JSON dict with the bot state as reported by the bot.
	//
	// Swarming itself mostly ignores this information, but it is exposed via API
	// and UI, allowing bots to report extended information about themselves to
	// Swarming clients.
	State []byte `gae:"state,noindex"`

	// SessionID is the current bot session ID reported when the bot connected.
	SessionID string `gae:"session_id,noindex"`

	// ExternalIP is the bot's IP address as seen by the server.
	ExternalIP string `gae:"external_ip,noindex"`

	// AuthenticatedAs is the bot's credentials as seen by the server.
	AuthenticatedAs identity.Identity `gae:"authenticated_as,noindex"`

	// Version of the bot code the bot is running.
	Version string `gae:"version,noindex"`

	// Quarantined means the bot is unhealthy and should not receive tasks.
	//
	// It is set when either:
	// - dimensions['quarantined'] or state['quarantined'] is set by the bot.
	// - API requests from the bot appear to be malformed.
	Quarantined bool `gae:"quarantined,noindex"`

	// Maintenance message if the bot is in maintenance.
	//
	// Maintenance state, just like quarantined state, means the bot should not
	// receive tasks. The difference is that maintenance is an expected condition:
	//   - The bot moves into maintenance state in expected moments.
	//   - It is expected to be short and end automatically.
	Maintenance string `gae:"maintenance_msg,noindex"`

	// TaskID is the packed TaskRunResult key of the relevant task, if any.
	//
	// For BotInfo, it identifies the current TaskRunResult being executed by
	// the bot.
	//
	// For BotEvent, it is relevant for event types `request_task`, `task_killed`,
	// `task_completed`, `task_error`.
	//
	// Note that it is **not** a packed TaskResultSummary. This `task_id` ends in
	// `1` instead of `0`.
	//
	// TODO(vadimsh): This is unfortunate, since this field ends up in BQ exports
	// where it causes confusion: task IDs in other BQ exports are "packed
	// TaskResultSummary ID", i.e. end in 0. This complicates joining BQ tables.
	TaskID string `gae:"task_id,noindex"`

	// LastSeen is the last time the bot contacted the server, if ever.
	//
	// Note that it is unindexed to avoid hotspotting the datastore, see
	// https://chromium.googlesource.com/infra/luci/luci-py/+/4e9aecba
	LastSeen datastore.Optional[time.Time, datastore.Unindexed] `gae:"last_seen_ts"`

	// IdleSince is when the bot became idle last time, if ever.
	//
	// It is unset when running the task or hooks.
	IdleSince datastore.Optional[time.Time, datastore.Unindexed] `gae:"idle_since_ts"`

	// ExpireAt is when this entity can be deleted via Cloud Datastore TTL policy.
	ExpireAt time.Time `gae:"expire_at,noindex"`

	// LegacyProperties is no longer used.
	LegacyLeaseID LegacyProperty `gae:"lease_id"`

	// LegacyLeaseExpiration is no longer used.
	LegacyLeaseExpiration LegacyProperty `gae:"lease_expiration_ts"`

	// LegacyLeasedIndefinitely is no longer used.
	LegacyLeasedIndefinitely LegacyProperty `gae:"leased_indefinitely"`

	// LegacyMachineType is no longer used.
	LegacyMachineType LegacyProperty `gae:"machine_type"`

	// LegacyMachineLease is no longer used.
	LegacyMachineLease LegacyProperty `gae:"machine_lease"`

	// LegacyStateJSON is no longer used.
	LegacyStateJSON LegacyProperty `gae:"state_json"`

	// LegacyDimensions is no longer used.
	LegacyDimensions LegacyProperty `gae:"dimensions"`

	// LegacyIsBusy is no longer used.
	LegacyIsBusy LegacyProperty `gae:"is_busy"`
}

BotCommon contains properties that are common to both BotInfo and BotEvent.

It is not meant to be stored in the datastore on its own, only as an embedded struct inside BotInfo or BotEvent.

type BotDimensions 

type BotDimensions map[string][]string

BotDimensions is a map with bot dimensions as `key => [values]`.

This type represents bot dimensions in the datastore as a JSON-encoded unindexed blob. There's an alternative "flat" indexed representation as a list of `key:value` pairs. It is used in BotCommon.Dimensions property.

func (*BotDimensions) FromProperty 

func (p *BotDimensions) FromProperty(prop datastore.Property) error

FromProperty loads a JSON-blob property.

func (*BotDimensions) ToProperty 

func (p *BotDimensions) ToProperty() (datastore.Property, error)

ToProperty stores the value as a JSON-blob property.

func (BotDimensions) ToProto 

func (p BotDimensions) ToProto() []*apipb.StringListPair

ToProto returns []apipb.StringListPair, sorted by keys.

func (BotDimensions) ToStructPB 

func (p BotDimensions) ToStructPB() *structpb.Struct

ToStructPB returns a structpb.Struct.

type BotEvent 

type BotEvent struct {
	BotCommon

	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the bot and this particular event.
	//
	// ID is auto-generated by the datastore. The bot is identified via the
	// parent key, which can be constructed via BotRootKey(...).
	Key *datastore.Key `gae:"$key"`

	// Timestamp of when this event happened.
	//
	// The index is used in a bunch of places:
	// 1. For ordering events chronologically when listing them.
	// 2. Pagination for BQ exports.
	// 3. Old event cleanup cron.
	Timestamp time.Time `gae:"ts"`

	// EventType describes what has happened.
	EventType BotEventType `gae:"event_type,noindex"`

	// Message is an optional free form message associated with the event.
	Message string `gae:"message,noindex"`

	// Dimensions is a sorted list of dimensions reported by the bot.
	//
	// TODO(vadimsh): Stop indexing this after turning down native Swarming
	// scheduler. This index is only used in has_capacity(...) implementation,
	// which is a part of the native Swarming scheduler and it not used when
	// running on top of RBE. This index is pretty big (~6 TB) and getting rid
	// of it may also speed up the bot event insertion transaction.
	Dimensions []string `gae:"dimensions_flat"`
}

BotEvent captures information about the bot during some state transition.

Entities of this kind are immutable. They essentially form a log with the bot history. Entries are indexed by the timestamp to allow querying this log in the chronological order.

func (*BotEvent) IsIdle 

func (e *BotEvent) IsIdle() bool

IsIdle is true if this event represents the bot being idle.

func (*BotEvent) QuarantineMessage 

func (e *BotEvent) QuarantineMessage() string

QuarantineMessage returns the explanation of why the bot is quarantined.

Returns an empty string if the state doesn't contain "quarantined" field.

func (*BotEvent) ToProto 

func (e *BotEvent) ToProto() *apipb.BotEventResponse

ToProto converts BotEvent to apipb.BotEventResponse.

type BotEventType 

type BotEventType string

BotEventType identifies various known bot events. 

const (
	BotEventConnected BotEventType = "bot_connected"
	BotEventError     BotEventType = "bot_error"
	BotEventIdle      BotEventType = "bot_idle"
	BotEventLog       BotEventType = "bot_log"
	BotEventMissing   BotEventType = "bot_missing"
	BotEventPolling   BotEventType = "bot_polling"
	BotEventRebooting BotEventType = "bot_rebooting"
	BotEventShutdown  BotEventType = "bot_shutdown"
	BotEventTerminate BotEventType = "bot_terminate"
)

Bot events that happen outside the scope of a task. 

const (
	BotEventRestart BotEventType = "request_restart"
	BotEventSleep   BotEventType = "request_sleep"
	BotEventTask    BotEventType = "request_task"
	BotEventUpdate  BotEventType = "request_update"
)

Bot events representing polling outcomes. 

const (
	BotEventTaskCompleted BotEventType = "task_completed"
	BotEventTaskError     BotEventType = "task_error"
	BotEventTaskKilled    BotEventType = "task_killed"
	BotEventTaskUpdate    BotEventType = "task_update"
)

Bot events related to running tasks.

type BotInfo 

type BotInfo struct {
	BotCommon

	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key is derived based on the bot ID, see BotInfoKey.
	Key *datastore.Key `gae:"$key"`

	// Dimensions is a sorted list of dimensions reported by the bot.
	//
	// Dimensions are used for task selection. They are encoded as a sorted list
	// of `key:value` strings. Keep in mind that the same key can be used
	// multiple times.
	//
	// The index is used to filter bots by their dimensions in bot listing API.
	Dimensions []string `gae:"dimensions_flat"`

	// Composite encodes the current state of the bot.
	//
	// For datastore performance reasons it encodes multiple aspects of the state
	// in a single indexed multi-valued field, resulting in a somewhat weird
	// semantics.
	//
	// The slice always have 4 items, with following meaning:
	//
	// Composite[0] is one of:
	//    BotStateInMaintenance    = 1 << 8  # 256
	//    BotStateNotInMaintenance = 1 << 9  # 512
	// Composite[1] is one of:
	//    BotStateDead  = 1 << 6  # 64
	//    BotStateAlive = 1 << 7  # 128
	// Composite[2] is one of:
	//    BotStateQuarantined = 1 << 2  # 4
	//    BotStateHealthy     = 1 << 3  # 8
	// Composite[3] is one of:
	//    BotStateBusy = 1 << 0  # 1
	//    BotStateIdle = 1 << 1  # 2
	Composite []BotStateEnum `gae:"composite"`

	// FirstSeen is when the bot was seen for the first time.
	FirstSeen time.Time `gae:"first_seen_ts,noindex"`

	// TaskName matches TaskRequest.Name of the task the the bot executes now.
	//
	// In other words its the title of the task identified by BotCommon.TaskID.
	// Empty if the bot is not executing any tasks now.
	TaskName string `gae:"task_name,noindex"`
}

BotInfo contains the latest information about a bot.

func (*BotInfo) BotID 

func (b *BotInfo) BotID() string

BotID extracts the bot ID from the entity key.

func (*BotInfo) DimensionsByKey 

func (b *BotInfo) DimensionsByKey(k string) (values []string)

DimensionsByKey returns a list of dimension values with the given key.

func (*BotInfo) GetStatus 

func (b *BotInfo) GetStatus() string

GetStatus returns the bot status.

func (*BotInfo) IsDead 

func (b *BotInfo) IsDead() bool

IsDead is true if this bot is considered dead.

func (*BotInfo) IsInMaintenance 

func (b *BotInfo) IsInMaintenance() bool

IsInMaintenance is true if this bot is in maintenance.

func (*BotInfo) ToProto 

func (b *BotInfo) ToProto() *apipb.BotInfo

ToProto converts BotInfo to apipb.BotInfo.

type BotRoot 

type BotRoot struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key is derived based on the bot ID, see BotRootKey.
	Key *datastore.Key `gae:"$key"`

	// LegacyCurrent is no longer used.
	LegacyCurrent LegacyProperty `gae:"current"`
}

BotRoot is an entity group root of entities representing a single bot.

Presence of this entity indicates there are BotEvent entities for this bot.

TODO(vadimsh): This entity is unnecessary complication. Old entities cleanup should happen via Cloud Datastore TTL feature, then this entity is not needed.

type BotStateEnum 

type BotStateEnum int64

BotStateEnum is used to represent state of the bot in datastore.

See comment for BotCommon.Composite. Individual values should not leak in any public APIs, it is an implementation detail. 

const (
	BotStateBusy             BotStateEnum = 1 << 0
	BotStateIdle             BotStateEnum = 1 << 1
	BotStateQuarantined      BotStateEnum = 1 << 2
	BotStateHealthy          BotStateEnum = 1 << 3
	BotStateUnused1          BotStateEnum = 1 << 4
	BotStateUnused2          BotStateEnum = 1 << 5
	BotStateDead             BotStateEnum = 1 << 6
	BotStateAlive            BotStateEnum = 1 << 7
	BotStateInMaintenance    BotStateEnum = 1 << 8
	BotStateNotInMaintenance BotStateEnum = 1 << 9
)

Possible categories of bot state.

type BotsDimensionsAggregation 

type BotsDimensionsAggregation struct {

	// Key is BotsDimensionsAggregationKey.
	Key *datastore.Key `gae:"$key"`
	// LastUpdate is when this entity changed the last time.
	LastUpdate time.Time `gae:",noindex"`

	// Dimensions is actual aggregated dimensions map.
	//
	// Stored internally as zstd-compressed proto (since the datastore library
	// compressed protos by default if they are larger than some threshold).
	Dimensions *internalmodelpb.AggregatedDimensions
	// contains filtered or unexported fields
}

BotsDimensionsAggregation stores all observed bot dimensions per pool.

It is updated by scan.BotsDimensionsAggregator.

type BotsDimensionsAggregationInfo 

type BotsDimensionsAggregationInfo struct {

	// Key is BotsDimensionsAggregationInfoKey.
	Key *datastore.Key `gae:"$key"`
	// LastUpdate is when BotsDimensionsAggregation changed the last time.
	LastUpdate time.Time `gae:",noindex"`
	// contains filtered or unexported fields
}

BotsDimensionsAggregationInfo contains info about BotsDimensionsAggregation.

It is tiny in comparison. Its LastUpdate field can be used to quickly check if the aggregated dimensions set has changed and needs to be reloaded. Both entities are always updated in the same transaction.

It is updated by scan.BotsDimensionsAggregator.

type BotsDimensionsCache 

type BotsDimensionsCache struct {
	// contains filtered or unexported fields
}

BotsDimensionsCache maintain an in-process cache of aggregated bot dimensions sets.

Since aggregated bot dimensions are used relatively infrequently, the cache is initialized and refreshed lazily on use (instead of e.g. periodically in background).

func (*BotsDimensionsCache) Get 

func (b *BotsDimensionsCache) Get(ctx context.Context) (*BotsDimensionsSets, error)

Get returns the aggregated bot dimensions, fetching or refreshing it if necessary.

All returned errors should be treated as transient datastore errors.

type BotsDimensionsSets 

type BotsDimensionsSets struct {
	// contains filtered or unexported fields
}

BotsDimensionsSets is a parsed representation of BotsDimensionsAggregation.

Given a list of pools it can return a set of all observed bot dimensions in these pools.

Immutable.

func NewBotsDimensionsSets 

func NewBotsDimensionsSets(perPool []*internalmodelpb.AggregatedDimensions_Pool, lastUpdate time.Time) *BotsDimensionsSets

NewBotsDimensionsSets constructs BotsDimensionsSets based on aggregated data.

Assumes dimensions values are sorted lists already, as produced by scan.BotsDimensionsAggregator.

func (*BotsDimensionsSets) DimensionsGlobally 

func (s *BotsDimensionsSets) DimensionsGlobally() *apipb.BotsDimensions

DimensionsGlobally returns a set of dimensions across all known pools.

func (*BotsDimensionsSets) DimensionsInPools 

func (s *BotsDimensionsSets) DimensionsInPools(pools []string) *apipb.BotsDimensions

DimensionsInPools returns a set of dimensions of bots in all given pools.

Unknown pools are considered empty.

type BuildTask 

type BuildTask struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the task.
	//
	// See BuildTaskKey.
	Key *datastore.Key `gae:"$key"`

	// BuildID is the Buildbucket build ID associated with the Swarming task.
	BuildID string `gae:"build_id,noindex"`

	// BuildbucketHost is the Buildbucket host that has the build.
	BuildbucketHost string `gae:"buildbucket_host,noindex"`

	// UpdateID is a monotonically increasing integer that is used to compare when
	// updates (state changes) have occurred. A timestamp measured in ms is used.
	UpdateID int64 `gae:"update_id,noindex"`

	// LatestTaskStatus is a the latest status sent to Buildbucket.
	//
	// It is a TaskRunResult.State, but will be converted to Buildbucket status
	// when sending out the update.
	LatestTaskStatus apipb.TaskState `gae:"latest_task_status,noindex"`

	// PubSubTopic is the pubsub topic name that will be used to send
	// UpdateBuildTask messages to Buildbucket.
	PubSubTopic string `gae:"pubsub_topic,noindex"`

	// BotDimensions are bot dimensions at the moment the bot claimed the task.
	//
	// The same as in TaskRunResult.BotDimensions. Stored here to avoid extra
	// datastore fetch when sending updates to Buildbucket.
	BotDimensions BotDimensions `gae:"bot_dimensions"`

	// LegacyTaskStatus is no longer used.
	LegacyTaskStatus LegacyProperty `gae:"task_status"`
}

BuildTask stores the Buildbucket related fields.

Present only if the corresponding TaskRequest's HasBuildTask is true.

func (*BuildTask) ToProto 

func (bt *BuildTask) ToProto(result *TaskResultSummary, target string) *bbpb.Task

ToProto returns the corresponding bbpb.Task proto.

The TaskResultSummary is used to populate status details.

type CASDigest 

type CASDigest struct {
	// Hash is blob's hash digest as a hex string.
	Hash string `gae:"hash"`
	// SizeBytes is the blob size.
	SizeBytes int64 `gae:"size_bytes"`
}

CASDigest represents an RBE-CAS blob's digest.

Is is a representation of build.bazel.remote.execution.v2.Digest. See https://github.com/bazelbuild/remote-apis/blob/77cfb44a88577a7ade5dd2400425f6d50469ec6d/build/bazel/remote/execution/v2/remote_execution.proto#L753-L791

func (*CASDigest) ToProto 

func (p *CASDigest) ToProto() *apipb.Digest

ToProto converts CASDigest to apipb.CASDigest.

type CASOperationStats 

type CASOperationStats struct {
	// DurationSecs is how long the operation ran.
	DurationSecs float64 `gae:"duration"`

	// InitialItems is the number of items in the cache before the operation.
	InitialItems int64 `gae:"initial_number_items"`

	// InitialSize is the total cache size before the operation.
	InitialSize int64 `gae:"initial_size"`

	// ItemsCold is a set of sizes of items that were downloaded or uploaded.
	//
	// It is encoded in a special way, see packedintset.Unpack.
	ItemsCold []byte `gae:"items_cold"`

	// ItemsHot is a set of sizes of items that were already present in the cache.
	//
	// It is encoded in a special way, see packedintset.Unpack.
	ItemsHot []byte `gae:"items_hot"`
}

CASOperationStats is performance stats of a CAS operation.

Stored as a unindexed subentity of PerformanceStats entity.

func (*CASOperationStats) IsEmpty 

func (p *CASOperationStats) IsEmpty() bool

IsEmpty is true if this struct is unpopulated.

func (*CASOperationStats) ToProto 

func (p *CASOperationStats) ToProto() (*apipb.CASOperationStats, error)

ToProto converts the CASOperationStats struct to apipb.CASOperationStats.

type CASReference 

type CASReference struct {
	// CASInstance is a full name of RBE-CAS instance.
	CASInstance string `gae:"cas_instance"`
	// Digest identifies the root tree to fetch.
	Digest CASDigest `gae:"digest,lsp"`
}

CASReference described where to fetch input files from.

func (*CASReference) ToProto 

func (p *CASReference) ToProto() *apipb.CASReference

ToProto converts CASReference to apipb.CASReference.

type CIPDInput 

type CIPDInput struct {
	// Server is URL of the CIPD server (including "https://" schema).
	Server string `gae:"server"`
	// ClientPackage defines a version of the CIPD client to use.
	ClientPackage CIPDPackage `gae:"client_package,lsp"`
	// Packages is a list of packages to install.
	Packages []CIPDPackage `gae:"packages,lsp"`
}

CIPDInput specifies which CIPD client and packages to install.

func (*CIPDInput) IsPopulated 

func (p *CIPDInput) IsPopulated() bool

IsPopulated returns true if the struct carries some data.

func (*CIPDInput) ToProto 

func (p *CIPDInput) ToProto() *apipb.CipdInput

ToProto converts CIPDInput to apipb.CIPDInput.

type CIPDPackage 

type CIPDPackage struct {
	// PackageName is a package name template (e.g. may include `${platform}`).
	PackageName string `gae:"package_name"`
	// Version is a package version to install.
	Version string `gae:"version"`
	// Path is a path relative to the task directory where to install the package.
	Path string `gae:"path"`
}

CIPDPackage defines a CIPD package to install into the task directory.

func (*CIPDPackage) ToProto 

func (p *CIPDPackage) ToProto() *apipb.CipdPackage

ToProto converts CIPDPackage to apipb.CipdPackage.

type CacheEntry 

type CacheEntry struct {
	// Name is a logical cache name.
	Name string `gae:"name"`
	// Path is where to mount it relative to the task root directory.
	Path string `gae:"path"`
}

CacheEntry describes a named cache that should be present on the bot.

func (*CacheEntry) ToProto 

func (p *CacheEntry) ToProto() *apipb.CacheEntry

ToProto converts CacheEntry to apipb.CacheEntry.

type Containment 

type Containment struct {
	ContainmentType           apipb.Containment_ContainmentType `gae:"containment_type"`
	LowerPriority             bool                              `gae:"lower_priority"`
	LimitProcesses            int64                             `gae:"limit_processes"`
	LimitTotalCommittedMemory int64                             `gae:"limit_total_committed_memory"`
}

Containment describes the task process containment.

func (*Containment) ToProto 

func (p *Containment) ToProto() *apipb.Containment

ToProto converts Containment struct to apipb.Containment

type Env 

type Env map[string]string

Env is a list of `(key, value)` pairs with environment variables to set.

Stored in JSON form in the datastore.

func (*Env) FromProperty 

func (p *Env) FromProperty(prop datastore.Property) error

FromProperty loads a JSON-blob property.

func (*Env) ToProperty 

func (p *Env) ToProperty() (datastore.Property, error)

ToProperty stores the value as a JSON-blob property.

func (*Env) ToProto 

func (p *Env) ToProto() []*apipb.StringPair

ToProto converts Env to []*apipb.StringPair

type EnvPrefixes 

type EnvPrefixes map[string][]string

EnvPrefixes is a list of `(key, []value)` pairs with env prefixes to add.

Stored in JSON form in the datastore.

func (*EnvPrefixes) FromProperty 

func (p *EnvPrefixes) FromProperty(prop datastore.Property) error

FromProperty loads a JSON-blob property.

func (*EnvPrefixes) ToProperty 

func (p *EnvPrefixes) ToProperty() (datastore.Property, error)

ToProperty stores the value as a JSON-blob property.

func (EnvPrefixes) ToProto 

func (p EnvPrefixes) ToProto() []*apipb.StringListPair

ToProto converts EnvPrefixes to []*apipb.StringListPair

type Filter 

type Filter struct {
	// contains filtered or unexported fields
}

Filter represents a filter over the space of ["key:value"] tags.

Conceptually it is a list of AND'ed together checks on values of tags. Each such check compares each value of some particular tag to a set of allowed values (often just one). The same tag key is allowed to show up more than once. In that case there will be more than one filter on values of this tag (see the example below).

In API this filter is encoded by a list of `key:val1|val2|val3` pairs, where keys are allowed to be repeated.

For example, this filter: 

["os:Linux", "os:Ubuntu", "zone:us-central|us-east"]

Will match entities with following tags: 

["os:Linux", "os:Ubuntu", "os:Ubuntu-20", "zone:us-central"]
["os:Linux", "os:Ubuntu", "os:Ubuntu-22", "zone:us-easy"]

But it will not match these entities: 

["os:Linux", "os:Debian", "zone:us-central"]
["os:Linux", "os:Ubuntu", "os:Ubuntu-22", "zone:us-west"]

func NewFilter 

func NewFilter(tags []*apipb.StringPair) (Filter, error)

NewFilter parses a list of `("key", "val1|val2|val2")` pairs.

Empty filter is possible (if `tags` are empty).

func NewFilterFromKV 

func NewFilterFromKV(tags []string) (Filter, error)

NewFilterFromKV is a variant of NewFilter that takes "k:v" pairs as input.

func (Filter) Apply 

func (f Filter) Apply(q *datastore.Query, field string, mode SplitMode) []*datastore.Query

Apply applies this filter to a query, returning (potentially) multiple queries.

Results of these queries must be merged locally (e.g. via datastore.RunMulti) to get the final filtered result.

`field` is the datastore entity field to apply the filter on. It should be a multi-valued field with values of form "key:value".

If the filter is empty, returns a list with the original query as is.

func (Filter) IsEmpty 

func (f Filter) IsEmpty() bool

IsEmpty is true if this filter doesn't filter anything.

func (Filter) Pools 

func (f Filter) Pools() []string

Pools is a list of all pools mentioned in the filter (if any).

func (Filter) SplitForQuery 

func (f Filter) SplitForQuery(mode SplitMode) []Filter

SplitForQuery splits this filter into several simpler filters that can be used in datastore queries, with their results merged.

The unsplit filter is generally too complex for the datastore query planner to handle using existing indexes (e.g. an index on `dimensions_flat` and a composite index on `(dimensions_flat, composite)` pair when used for BotInfo queries).

Unfortunately due to datastore limits we can't just add all necessary composite indexes (like `(dimensions_flat, dimensions_flat, composite)` one). Since `dimensions_flat` is a repeated property, this results in too many indexed permutations of values, blowing up this index. Possible workarounds require changing the layout of BotInfo entities in datastore, but that would require imposing limits on public Swarming API (basically, we'll need to predefine what dimension keys are worth indexing and what are not; currently all are indexed).

Instead we split the query into N subqueries, run them in parallel and merge results locally. This is relatively expensive and scales poorly, but we need to do that only for complex queries that use multiple OR property filters. They are relatively rare.

If the original filter is empty, returns one empty filter as the output.

type LegacyBootstrapSecret 

type LegacyBootstrapSecret struct {

	// Key should be LegacyBootstrapSecretKey(...).
	Key *datastore.Key `gae:"$key"`
	// Values is a list of historical values of the secret.
	Values [][]byte `gae:"values,noindex"`
	// contains filtered or unexported fields
}

LegacyBootstrapSecret is a subset of the AuthSecret python entity with the bootstrap token secret.

type LegacyProperty 

type LegacyProperty struct{}

LegacyProperty is a placeholder for "recognizing" known legacy properties.

Properties of this type are silently discarded when read (and consequently not stored back when written). This is useful for dropping properties that were known to exist at some point, but which are no longer used by anything at all. If we just ignore them completely, they'll end up in `Extra` maps, which we want to avoid (`Extra` is only for truly unexpected properties).

func (*LegacyProperty) FromProperty 

func (*LegacyProperty) FromProperty(p datastore.Property) error

FromProperty implements datastore.PropertyConverter.

func (*LegacyProperty) ToProperty 

func (*LegacyProperty) ToProperty() (datastore.Property, error)

ToProperty implements datastore.PropertyConverter.

type OperationStats 

type OperationStats struct {
	// DurationSecs is how long the operation ran.
	DurationSecs float64 `gae:"duration"`
}

OperationStats is performance stats of a particular operation.

Stored as a unindexed subentity of PerformanceStats entity.

func (*OperationStats) ToProto 

func (p *OperationStats) ToProto() *apipb.OperationStats

ToProto converts the OperationStats struct to apipb.OperationStats.

type PerformanceStats 

type PerformanceStats struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the task, see PerformanceStatsKey.
	Key *datastore.Key `gae:"$key"`

	// BotOverheadSecs is the total overhead in seconds, summing overhead from all
	// sections defined below.
	BotOverheadSecs float64 `gae:"bot_overhead,noindex"`

	// CacheTrim is stats of cache trimming before the dependency installations.
	CacheTrim OperationStats `gae:"cache_trim,lsp,noindex"`

	// PackageInstallation is stats of installing CIPD packages before the task.
	PackageInstallation OperationStats `gae:"package_installation,lsp,noindex"`

	// NamedCachesInstall is stats of named cache mounting before the task.
	NamedCachesInstall OperationStats `gae:"named_caches_install,lsp,noindex"`

	// NamedCachesUninstall is stats of named cache unmounting after the task.
	NamedCachesUninstall OperationStats `gae:"named_caches_uninstall,lsp,noindex"`

	// IsolatedDownload is stats of CAS dependencies download before the task.
	IsolatedDownload CASOperationStats `gae:"isolated_download,lsp,noindex"`

	// IsolatedUpload is stats of CAS uploading operation after the task.
	IsolatedUpload CASOperationStats `gae:"isolated_upload,lsp,noindex"`

	// Cleanup is stats of work directory cleanup operation after the task.
	Cleanup OperationStats `gae:"cleanup,lsp,noindex"`
}

PerformanceStats contains various timing and performance information about the task as reported by the bot.

func (*PerformanceStats) ToProto 

func (p *PerformanceStats) ToProto() (*apipb.PerformanceStats, error)

ToProto converts the PerformanceStats struct to apipb.PerformanceStats.

type PerformanceStatsFetcher 

type PerformanceStatsFetcher struct {
	// contains filtered or unexported fields
}

PerformanceStatsFetcher can fetch PerformanceStats entities in parallel.

It is a wrapper over generic BatchFetcher.

func NewPerformanceStatsFetcher 

func NewPerformanceStatsFetcher(ctx context.Context) *PerformanceStatsFetcher

NewPerformanceStatsFetcher constructs a new fetcher.

It should be closed with Close to avoid leaking goroutines.

func (*PerformanceStatsFetcher) Close 

func (f *PerformanceStatsFetcher) Close()

Close makes sure all internal goroutines are canceled and stopped.

func (*PerformanceStatsFetcher) Fetch 

func (f *PerformanceStatsFetcher) Fetch(ctx context.Context, task *apipb.TaskResultResponse, key PerformanceStatsHolder)

Fetch starts fetching PerformanceStats of the task if it has them.

func (*PerformanceStatsFetcher) Finish 

func (f *PerformanceStatsFetcher) Finish(tasks []*apipb.TaskResultResponse) error

Finish waits for all fetches to finish and populates PerformanceStats fields.

Updates protos in `tasks` in-place.

type PerformanceStatsHolder 

type PerformanceStatsHolder interface {
	PerformanceStatsKey(ctx context.Context) *datastore.Key
}

PerformanceStatsHolder provides the PerformanceStats entity key.

It is implemented by TaskRunResult and TaskResultSummary.

type ResultDBConfig 

type ResultDBConfig struct {
	// Enable indicates if the task should have ResultDB invocation.
	//
	// If True and this task is not deduplicated, create
	// "task-{swarming_hostname}-{run_id}" invocation for this task, provide its
	// update token to the task subprocess via LUCI_CONTEXT and finalize the
	// invocation when the task is done.
	//
	// If the task is deduplicated, then TaskResult.InvocationName will be the
	// invocation name of the original task.
	Enable bool `gae:"enable"`
}

ResultDBConfig is ResultDB integration configuration for a task.

func (*ResultDBConfig) ToProto 

func (p *ResultDBConfig) ToProto() *apipb.ResultDBCfg

ToProto converts ResultDBConfig struct to apipb.ResultDBCfg.

type ResultDBInfo 

type ResultDBInfo struct {
	// Hostname is the ResultDB service hostname e.g. "results.api.cr.dev".
	Hostname string `gae:"hostname"`
	// Invocation is the ResultDB invocation name for the task, if any.
	Invocation string `gae:"invocation"`
}

ResultDBInfo contains invocation ID of the task.

func (*ResultDBInfo) ToProto 

func (p *ResultDBInfo) ToProto() *apipb.ResultDBInfo

ToProto converts ResultDBInfo to an apipb.ResultDBInfo.

type SecretBytes 

type SecretBytes struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the task and its concrete slice.
	//
	// See SecretBytesKey.
	Key *datastore.Key `gae:"$key"`

	// SecretBytes is the actual secret bytes blob.
	SecretBytes []byte `gae:"secret_bytes,noindex"`
}

SecretBytes defines an optional secret byte string logically defined within TaskRequest.

Stored separately for size and data-leakage reasons. All task slices reuse the same secret bytes (which is an implementation artifact, not a desired property). If a task slice uses secret bytes, it has HasSecretBytes == true.

type SplitMode 

type SplitMode int

SplitMode is a parameter for SplitForQuery and Apply methods. 

const (
	// SplitOptimally indicates to make as few split as possible.
	//
	// Some queries may end up using "OR" filters, but no more than one such
	// filter per query. Such queries are still accepted by the datastore.
	SplitOptimally SplitMode = 0

	// SplitCompletely indicates to split a filter into elementary filters.
	//
	// Elementary filters do not have "OR" in them. This is used in testing to
	// cover code paths that merge results of multiple queries. This is needed
	// because the local testing environment current (as of Jan 2024) doesn't
	// actually support OR queries at all.
	SplitCompletely SplitMode = 1
)

func FilterTasksByState 

func FilterTasksByState(q *datastore.Query, state apipb.StateQuery, splitMode SplitMode) ([]*datastore.Query, SplitMode)

FilterTasksByState limits a TaskResultSummary query to return tasks in particular state.

For StateQuery_QUERY_PENDING_RUNNING filter, depending on passed SplitMode, may either split the query into multiple queries that need to run in parallel, or append an "IN" filter to the original query. For all other state filters just adds simple "EQ" query filters.

Per current datastore limitations (see Filter.SplitForQuery), a query can have at most one "IN" filter. Thus if FilterTasksByState returns a query with an "IN" filter, all queries built on top of it must not add any more "IN" filters. This is communicated by returning SplitCompletely SplitMode that should be applied to all subsequent splittings (if any).

type StateFilter 

type StateFilter struct {
	// Quarantined filters bots based on whether they are quarantined.
	Quarantined apipb.NullableBool
	// InMaintenance filters bots based on whether they are in maintenance mode.
	InMaintenance apipb.NullableBool
	// IsDead filters bots based on whether they are connected or not.
	IsDead apipb.NullableBool
	// IsBusy filters bots based on whether they execute any task or not.
	IsBusy apipb.NullableBool
}

StateFilter represents a filter over the possible bot states.

Each field is a filter on one aspect of the bot state with possible values being TRUE (meaning "yes"), FALSE (meaning "no") and NULL (meaning "don't care").

type TaskDimensions 

type TaskDimensions map[string][]string

TaskDimensions defines requirements for a bot to match a task.

Stored in JSON form in the datastore.

func (*TaskDimensions) FromProperty 

func (p *TaskDimensions) FromProperty(prop datastore.Property) error

FromProperty loads a JSON-blob property.

func (TaskDimensions) Hash 

func (p TaskDimensions) Hash() uint32

Hash returns a 32 bits unsigned int that is a hash of TaskDimensions.

Dimensions values still have "|" in them, i.e. this is calculated prior to expansion of OR-ed dimensions into a disjunction of dimension sets.

Dimensions values are expected to be sorted, and it's also expected the elements in OR-ed dimensions to be sorted.

The return value is guaranteed to be a non-zero int so it can be used as an entity ID.

func (*TaskDimensions) ToProperty 

func (p *TaskDimensions) ToProperty() (datastore.Property, error)

ToProperty stores the value as a JSON-blob property.

func (TaskDimensions) ToProto 

func (p TaskDimensions) ToProto() []*apipb.StringPair

ToProto converts TaskDimensions to []*apipb.StringPair

type TaskIDVariant 

type TaskIDVariant int

TaskIDVariant is an enum with possible variants of task ID encoding. 

const (
	// AsRequest instructs RequestKeyToTaskID to produce an ID ending with `0`.
	AsRequest TaskIDVariant = 0
	// AsRunResult instructs RequestKeyToTaskID to produce an ID ending with `1`.
	AsRunResult TaskIDVariant = 1
)

type TaskOutputChunk 

type TaskOutputChunk struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the task and the chunk index, see TaskOutputChunkKey.
	Key *datastore.Key `gae:"$key"`

	// Chunk is zlib-compressed chunk of the output.
	Chunk []byte `gae:"chunk,noindex"`

	// Gaps is a series of 2 integer pairs, which specifies the part that are
	// invalid. Normally it should be empty. All values are relative to the start
	// of this chunk offset.
	Gaps []int32 `gae:"gaps,noindex"`
}

TaskOutputChunk represents a chunk of the task console output.

The number of such chunks per task is tracked in TaskRunResult.StdoutChunks. Each entity holds a compressed segment of the task output. For all entities except the last one, the uncompressed size of the segment is ChunkSize. That way it is always possible to figure out what chunk to use for a given output offset.

type TaskProperties 

type TaskProperties struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Idempotent, if true, means it is OK to skip running this task if there's
	// already a successful task with the same properties hash.
	//
	// The results of such previous task will be reused as results of this task.
	Idempotent bool `gae:"idempotent"`

	// Dimensions are used to match this task to a bot.
	//
	// This is conceptually a set of `(key, value1 | value2 | ...)` pairs, each
	// defining some constraint on a matching bot. For a bot to match the task,
	// it should satisfy all constraints.
	//
	// For a bot to match a single `(key, value1 | value2| ...)` constraint, bot's
	// value for dimension `key` should be equal to `value1` or `value2` and so
	// on.
	Dimensions TaskDimensions `gae:"dimensions"`

	// ExecutionTimeoutSecs is the maximum duration the bot can take to run this
	// task.
	//
	// It's also known as `hard_timeout` in the bot code.
	ExecutionTimeoutSecs int64 `gae:"execution_timeout_secs"`

	// GracePeriodSecs is the time between sending SIGTERM and SIGKILL when the
	// task times out.
	//
	// As soon as the ask reaches its execution timeout, the task process is sent
	// SIGTERM. The process should clean up and terminate. If it is still running
	// after GracePeriodSecs, it gets killed via SIGKILL.
	GracePeriodSecs int64 `gae:"grace_period_secs"`

	// IOTimeoutSecs controls how soon to consider a "silent" process to be stuck.
	//
	// If a subprocess doesn't output new data to stdout for  IOTimeoutSecs,
	// consider the task timed out. Optional.
	IOTimeoutSecs int64 `gae:"io_timeout_secs"`

	// Command is a command line to run.
	Command []string `gae:"command"`

	// RelativeCwd is a working directory relative to the task root to run
	// the command in.
	RelativeCwd string `gae:"relative_cwd"`

	// Env is environment variables to set when running the task process.
	Env Env `gae:"env"`

	// EnvPrefixes is environment path prefix variables.
	//
	// E.g. if a `PATH` key has values `[a, b]`, then the final `PATH` env var
	// will be `a;b;$PATH` (where `;` is a platforms' env path separator).
	EnvPrefixes EnvPrefixes `gae:"env_prefixes"`

	// Caches defines what named caches to mount.
	Caches []CacheEntry `gae:"caches,lsp"`

	// CASInputRoot is a digest of the input root uploaded to RBE-CAS.
	//
	// This MUST be digest of `build.bazel.remote.execution.v2.Directory`.
	CASInputRoot CASReference `gae:"cas_input_root,lsp"`

	// CIPDInput defines what CIPD packages to install.
	CIPDInput CIPDInput `gae:"cipd_input,lsp"`

	// Outputs is a list of extra outputs to upload to RBE-CAS as task results.
	//
	// If empty, only files written to `${ISOLATED_OUTDIR}` will be returned.
	// Otherwise, the files in this list will be added to those in that directory.
	Outputs []string `gae:"outputs"`

	// HasSecretBytes, if true, means there's a SecretBytes entity associated with
	// the parent TaskRequest.
	HasSecretBytes bool `gae:"has_secret_bytes"`

	// Containment defines what task process containment mechanism to use.
	//
	// Not really implemented currently.
	Containment Containment `gae:"containment,lsp"`

	// LegacyInputsRef is no longer used.
	LegacyInputsRef LegacyProperty `gae:"inputs_ref"`
}

TaskProperties defines where and how to run the task.

This entity is not saved in the DB as a standalone entity, instead it is embedded in a TaskSlice, unindexed.

This entity is immutable.

func (*TaskProperties) ToProto 

func (p *TaskProperties) ToProto() *apipb.TaskProperties

ToProto converts TaskProperties to apipb.TaskProperties.

type TaskRequest 

type TaskRequest struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key is derived based on time and randomness.
	//
	// It is normally serialized into a hex string. See TaskRequestKey.
	Key *datastore.Key `gae:"$key"`

	// TxnUUID is used internally to make the transaction that creates TaskRequest
	// idempotent.
	//
	// Just a randomly generated string. Should not be used for anything else.
	// Should not show up anywhere.
	TxnUUID string `gae:"txn_uuid,noindex"`

	// TaskSlices defines what to run.
	//
	// Each slice defines what to run and where. Slices are attempted one after
	// another, until some ends up running on a bot. If an attempt to schedule
	// a particular slice fails (i.e. there are no bots matching requested
	// dimensions or the slice sits queued for too long and expires), the next
	// slice is scheduled in its place.
	//
	// This is primarily used to requests bots with "hot" caches before falling
	// back on more generic bots.
	TaskSlices []TaskSlice `gae:"task_slices,lsp,noindex"`

	// Created is a precise timestamp when this request was registered.
	//
	// Unlike the timestamp in the entity key, this one has microsecond precision.
	//
	// The index is used in BQ exports and when cleaning up old tasks.
	//
	// TODO(vadimsh): BQ export can be switched to use __key__ and old tasks
	// should be cleaned up via a TTL policy. Then we can remove this index.
	Created time.Time `gae:"created_ts"`

	// Expiration is when to give up trying to run the task.
	//
	// If the task request is not scheduled by this moment, it will be aborted
	// with EXPIRED status. This value always matches Expiration of the last
	// TaskSlice.
	//
	// TODO(vadimsh): Why is it stored separately at all?
	Expiration time.Time `gae:"expiration_ts,noindex"`

	// Name of this task request as provided by the caller. Only for description.
	Name string `gae:"name,noindex"`

	// ParentTaskID is set when this task was created from another task.
	//
	// This is packed TaskToRun ID of an attempt that launched this task or null
	// if this task doesn't have a parent.
	//
	// The index is used to find children of a particular parent task to cancel
	// them when the parent task dies.
	ParentTaskID datastore.Nullable[string, datastore.Indexed] `gae:"parent_task_id"`

	// RootTaskID identifies the task run that started the tree of Swarming tasks.
	//
	// If a new task doesn't have a parent, this is unset. Otherwise if the parent
	// task has RootTaskID, this value is used in the new task. Otherwise
	// ParentTaskID itself is used.
	//
	// That way all tasks from the same task tree (except the root one itself)
	// will have RootTaskID populated.
	//
	// This is used in BQ exported. Not clear if anyone actually consumes this
	// information.
	RootTaskID string `gae:"root_task_id,noindex"`

	// Authenticated is an identity that triggered this task.
	//
	// Derived from the caller credentials.
	Authenticated identity.Identity `gae:"authenticated,noindex"`

	// What user to "blame" for this task.
	//
	// Can be arbitrary, not asserted by any credentials.
	User string `gae:"user,noindex"`

	// Tags classify this task in some way.
	//
	// This is a generated property. This property contains both the tags
	// specified by the user and the tags from every TaskSlice.
	Tags []string `gae:"tags,noindex"`

	// ManualTags are tags that are provided by the user.
	//
	// This is used to regenerate the list of tags for TaskResultSummary based on
	// the actual TaskSlice used.
	ManualTags []string `gae:"manual_tags,noindex"`

	// ServiceAccount indicates what credentials the task uses when calling other
	// services.
	//
	// Possible values are: `none`, `bot` or `<email>`.
	ServiceAccount string `gae:"service_account,noindex"`

	// Realm is task's realm controlling who can see and cancel this task.
	//
	// Missing for internally generated tasks such as termination tasks.
	Realm string `gae:"realm,noindex"`

	// RealmsEnabled is a legacy flag that should always be True.
	//
	// TODO(vadimsh): Get rid of it when Python code is no more.
	RealmsEnabled bool `gae:"realms_enabled,noindex"`

	// SchedulingAlgorithm is a scheduling algorithm set in pools.cfg at the time
	// the request was created.
	//
	// TODO(vadimsh): This does nothing for RBE pools.
	SchedulingAlgorithm configpb.Pool_SchedulingAlgorithm `gae:"scheduling_algorithm,noindex"`

	// Priority of the this task.
	//
	// A lower number means higher priority.
	Priority int64 `gae:"priority,noindex"`

	// BotPingToleranceSecs is a maximum delay between bot pings before the bot is
	// considered dead while running a task.
	//
	// TODO(vadimsh): Why do this per-task instead of per-pool or even hardcoded?
	BotPingToleranceSecs int64 `gae:"bot_ping_tolerance_secs,noindex"`

	// RBEInstance is an RBE instance to send the task to or "" to use Swarming
	// native scheduler.
	//
	// Initialized when creating a task based on pools.cfg config.
	RBEInstance string `gae:"rbe_instance,noindex"`

	// PubSubTopic is a topic to send a task completion notification to.
	PubSubTopic string `gae:"pubsub_topic,noindex"`

	// PubSubAuthToken is a secret token to send as `auth_token` PubSub message
	// attribute.
	PubSubAuthToken string `gae:"pubsub_auth_token,noindex"`

	// PubSubUserData is data to send in `userdata` field of PubSub messages.
	PubSubUserData string `gae:"pubsub_userdata,noindex"`

	// ResultDBUpdateToken is the ResultDB invocation's update token for the task
	// run that was created for this request.
	//
	// This is empty if the task was deduplicated or if ResultDB integration was
	// not enabled for this task.
	ResultDBUpdateToken string `gae:"resultdb_update_token,noindex"`

	// ResultDB is ResultDB integration configuration for this task.
	ResultDB ResultDBConfig `gae:"resultdb,lsp,noindex"`

	// HasBuildTask is true if the TaskRequest has an associated BuildTask.
	HasBuildTask bool `gae:"has_build_task,noindex"`

	// LegacyProperties is no longer used.
	LegacyProperties LegacyProperty `gae:"properties"`

	// LegacyHasBuildToken is no longer used.
	LegacyHasBuildToken LegacyProperty `gae:"has_build_token"`
}

TaskRequest contains a user request to execute a task.

Key ID is a decreasing integer based on time plus some randomness on lower order bits. See NewTaskRequestID for the complete gory details.

This entity is immutable.

func (*TaskRequest) BotID 

func (p *TaskRequest) BotID() string

BotID is a specific bot the task wants to run on, if any.

func (*TaskRequest) Pool 

func (p *TaskRequest) Pool() string

Pool is the pool the task wants to run in.

func (*TaskRequest) TaskAuthInfo 

func (p *TaskRequest) TaskAuthInfo(ctx context.Context) (*acls.TaskAuthInfo, error)

TaskAuthInfo returns information about the task for ACL checks.

This implements acls.Task.

func (*TaskRequest) ToProto 

func (p *TaskRequest) ToProto() *apipb.TaskRequestResponse

ToProto converts a TaskRequest to apipb.TaskRequestResponse.

type TaskRequestID 

type TaskRequestID struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key is derived from the request ID.
	//
	// See TaskRequestIDKey.
	Key *datastore.Key `gae:"$key"`

	// TaskID is a packed TaskResultSummary key identifying TaskRequest matching
	// this request ID.
	//
	// Use TaskRequestKey(...) to get the actual datastore key from it.
	TaskID string `gae:"task_id,noindex"`

	// ExpireAt is when this entity should be removed from the datastore.
	//
	// This is used by a TTL policy: https://cloud.google.com/datastore/docs/ttl
	ExpireAt time.Time `gae:"expire_at,noindex"`
}

TaskRequestID defines a mapping between request's idempotency ID and task ID.

It is a root-level entity. Used to make sure at most one TaskRequest entity is created per the given request ID.

type TaskResultCommon 

type TaskResultCommon struct {
	// State is the current state of the task.
	//
	// The index is used in task listing queries to filter by.
	State apipb.TaskState `gae:"state"`

	// Modified is when this entity was created or last modified.
	Modified time.Time `gae:"modified_ts,noindex"`

	// BotVersion is a version of the bot code running the task, if already known.
	BotVersion string `gae:"bot_version,noindex"`

	// BotDimensions are bot dimensions at the moment the bot claimed the task.
	BotDimensions BotDimensions `gae:"bot_dimensions"`

	// BotIdleSince is when the bot running this task finished its previous task,
	// if already known.
	BotIdleSince datastore.Optional[time.Time, datastore.Unindexed] `gae:"bot_idle_since_ts"`

	// BotLogsCloudProject is a GCP project where the bot uploads its logs.
	BotLogsCloudProject string `gae:"bot_logs_cloud_project,noindex"`

	// ServerVersions is a set of server version(s) that touched this entity.
	ServerVersions []string `gae:"server_versions,noindex"`

	// CurrentTaskSlice is an index of the active task slice in the TaskRequest.
	//
	// For TaskResultSummary it is the currently running slice (and it changes
	// over lifetime of a task if it has many slices). For TaskRunResult it is the
	// index representing this concrete run.
	CurrentTaskSlice int64 `gae:"current_task_slice,noindex"`

	// Started is when the bot started executing the task.
	//
	// The index is used in task listing queries to order by this property.
	Started datastore.Nullable[time.Time, datastore.Indexed] `gae:"started_ts"`

	// Completed is when the task switched into a final state.
	//
	// This is either when the bot finished executing it, or when it expired or
	// was canceled.
	//
	// The index is used in a bunch of places:
	//   1. In task listing queries to order by this property.
	//   2. In crashed bot detection to list still pending or running tasks.
	//   3. In BQ export pagination.
	Completed datastore.Nullable[time.Time, datastore.Indexed] `gae:"completed_ts"`

	// Abandoned is set when a task had an internal failure, timed out or was
	// killed by a client request.
	//
	// The index is used in task listing queries to order by this property.
	Abandoned datastore.Nullable[time.Time, datastore.Indexed] `gae:"abandoned_ts"`

	// DurationSecs is how long the task process was running.
	//
	// This is reported explicitly by the bot and it excludes all overhead.
	DurationSecs datastore.Optional[float64, datastore.Unindexed] `gae:"durations"` // legacy plural property name

	// ExitCode is the task process exit code for tasks in COMPLETED state.
	ExitCode datastore.Optional[int64, datastore.Unindexed] `gae:"exit_codes"` // legacy plural property name

	// Failure is true if the task finished with a non-zero process exit code.
	//
	// The index is used in task listing queries to filter by failure.
	Failure bool `gae:"failure"`

	// InternalFailure is true if the task failed due to an internal error.
	InternalFailure bool `gae:"internal_failure,noindex"`

	// StdoutChunks is the number of TaskOutputChunk entities with the output.
	StdoutChunks int64 `gae:"stdout_chunks,noindex"`

	// CASOutputRoot is the digest of the output root uploaded to RBE-CAS.
	CASOutputRoot CASReference `gae:"cas_output_root,lsp,noindex"`

	// CIPDPins is resolved versions of all the CIPD packages used in the task.
	CIPDPins CIPDInput `gae:"cipd_pins,lsp,noindex"`

	// MissingCIPD is the missing CIPD packages in CLIENT_ERROR state.
	MissingCIPD []CIPDPackage `gae:"missing_cipd,lsp,noindex"`

	// MissingCAS is the missing CAS digests in CLIENT_ERROR state.
	MissingCAS []CASReference `gae:"missing_cas,lsp,noindex"`

	// ResultDBInfo is ResultDB invocation info for this task.
	//
	// If this task was deduplicated, this contains invocation info from the
	// previously ran the task deduplicated from.
	ResultDBInfo ResultDBInfo `gae:"resultdb_info,lsp,noindex"`

	// LegacyOutputsRef is no longer used.
	LegacyOutputsRef LegacyProperty `gae:"outputs_ref"`
}

TaskResultCommon contains properties that are common to both TaskRunResult and TaskResultSummary.

It is not meant to be stored in the datastore on its own, only as an embedded struct inside TaskRunResult or TaskResultSummary.

type TaskResultSummary 

type TaskResultSummary struct {
	// TaskResultCommon embeds most of result properties.
	TaskResultCommon

	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the task, see TaskResultSummaryKey().
	Key *datastore.Key `gae:"$key"`

	// BotID is ID of a bot that runs this task, if already known.
	BotID datastore.Optional[string, datastore.Unindexed] `gae:"bot_id"`

	// Created is a precise timestamp when the task was submitted.
	//
	// Unlike the timestamp in the entity key, this one has microsecond precision.
	Created time.Time `gae:"created_ts,noindex"`

	// Tags are copied from the corresponding TaskRequest entity.
	//
	// The index is used in global task listing queries to filter by.
	Tags []string `gae:"tags"`

	// RequestName is copied from the corresponding TaskRequest entity.
	RequestName string `gae:"name,noindex"`

	// RequestUser is copied from the corresponding TaskRequest entity.
	RequestUser string `gae:"user,noindex"`

	// RequestPriority is copied from the corresponding TaskRequest entity.
	RequestPriority int64 `gae:"priority,noindex"`

	// RequestAuthenticated is copied from the corresponding TaskRequest entity.
	RequestAuthenticated identity.Identity `gae:"request_authenticated,noindex"`

	// RequestRealm is copied from the corresponding TaskRequest entity.
	RequestRealm string `gae:"request_realm,noindex"`

	// RequestPool is a pool the task is targeting.
	//
	// Derived from dimensions in the corresponding TaskRequest entity.
	RequestPool string `gae:"request_pool,noindex"`

	// RequestBotID is a concrete bot the task is targeting (if any).
	//
	// Derived from dimensions in the corresponding TaskRequest entity.
	RequestBotID string `gae:"request_bot_id,noindex"`

	// PropertiesHash is used to find duplicate tasks.
	//
	// It is set to a hash of properties of the task slice only when these
	// conditions are met:
	// - TaskSlice.TaskProperties.Idempotent is true.
	// - State is COMPLETED.
	// - Failure is false (i.e. ExitCode is 0).
	// - InternalFailure is false.
	//
	// The index is used to find an an existing duplicate task when submitting
	// a new task.
	PropertiesHash datastore.Optional[[]byte, datastore.Indexed] `gae:"properties_hash"`

	// TryNumber indirectly identifies if this task was dedupped.
	//
	// This field is leftover from when swarming had internal retries and for that
	// reason has a weird semantics. See https://crbug.com/1065101.
	//
	// Possible values:
	//   null: if the task is still pending, has expired or was canceled.
	//   1: if the task was assigned to a bot and either currently runs or has
	//      finished or crashed already.
	//   0: if the task was dedupped.
	//
	// The index is used in global task listing queries to find what tasks were
	// dedupped.
	TryNumber datastore.Nullable[int64, datastore.Indexed] `gae:"try_number"`

	// CostUSD is an approximate bot time cost spent executing this task.
	CostUSD float64 `gae:"costs_usd,noindex"` // legacy plural property name

	// CostSavedUSD is an approximate cost saved by deduping this task.
	CostSavedUSD float64 `gae:"cost_saved_usd,noindex"`

	// DedupedFrom is set if this task reused results of some previous task.
	//
	// See PropertiesHash for conditions when this is possible.
	//
	// DedupedFrom is a packed TaskRunResult key of the reused result. Note that
	// there's no TaskRunResult representing this task, since it didn't run.
	DedupedFrom string `gae:"deduped_from,noindex"`

	// ExpirationDelay is a delay from TaskRequest.ExpirationTS to the actual
	// expiry time.
	//
	// This is set at expiration process if the last task slice expired by
	// reaching its deadline. Unset if the last slice expired because there were
	// no bots that could run it.
	//
	// Exclusively for monitoring.
	ExpirationDelay datastore.Optional[float64, datastore.Unindexed] `gae:"expiration_delay"`
}

TaskResultSummary represents the overall result of a task.

Parent is a TaskRequest. Key id is always 1.

It is created (in PENDING state) as soon as the task is created and then mutated whenever the task changes its state. Its used by various task listing APIs.

func (*TaskResultSummary) CostsUSD 

func (p *TaskResultSummary) CostsUSD() []float32

CostsUSD converts the costUSD in TaskResultSummary to a []float32 containing that costUSD.

func (*TaskResultSummary) GetOutput 

func (p *TaskResultSummary) GetOutput(ctx context.Context, offset, length int64) ([]byte, error)

GetOutput returns the stdout content for the task.

Returns at most `length` bytes (perhaps less if reading the end of the output). Returns empty output if the task hasn't run.

func (*TaskResultSummary) IsActive 

func (p *TaskResultSummary) IsActive() bool

IsActive returns whether the task is either PENDING or RUNNING.

func (*TaskResultSummary) PendingNow 

func (p *TaskResultSummary) PendingNow(ctx context.Context, now time.Time) (diff time.Duration, deduped bool)

PendingNow returns the duration the task spent pending to be scheduled as of now.

If the duration is less than 0, return 0. If the task is deduped, return 0 and deduped as true.

func (*TaskResultSummary) PerformanceStatsKey 

func (p *TaskResultSummary) PerformanceStatsKey(ctx context.Context) *datastore.Key

PerformanceStatsKey returns PerformanceStats entity key or nil.

It returns a non-nil key if the performance stats entity can **potentially** exist (e.g. the task has finished running and reported its stats). It returns nil if the performance stats entity definitely doesn't exist.

If the task was dedupped, returns the key of the performance stats of the original task that actually ran.

func (*TaskResultSummary) TaskAuthInfo 

func (p *TaskResultSummary) TaskAuthInfo(ctx context.Context) (*acls.TaskAuthInfo, error)

TaskAuthInfo returns information about the task for ACL checks.

This implements acls.Task.

If the entity is very old (older than ~Oct 2023) and doesn't yet have RequestXXX fields populated, values are fetched from the corresponding TaskRequest entity. This should be rare and this code path is intentionally not optimized to avoid adding complexity.

The fallback can be removed ~May 2025 (when all "very old" entities will expire and disappear from the datastore).

See https://chromium.googlesource.com/infra/luci/luci-py/+/b50d4ba949cb25b7

func (*TaskResultSummary) TaskRequestKey 

func (p *TaskResultSummary) TaskRequestKey() *datastore.Key

TaskRequestKey returns the parent task request key or panics if it is unset.

func (*TaskResultSummary) TaskRunID 

func (p *TaskResultSummary) TaskRunID() string

TaskRunID returns the packed TaskRunResult key of the actual execution.

If the task was dedupped, it will be the key of the run that actually happened.

It is empty string if the task hasn't run (e.g. still pending or has expired).

func (*TaskResultSummary) ToProto 

func (p *TaskResultSummary) ToProto() *apipb.TaskResultResponse

ToProto converts the TaskResultSummary struct to an apipb.TaskResultResponse.

Note: This function will not handle PerformanceStats due to the requirement of fetching another datastore entity. Please refer to PerformanceStats to fetch them.

type TaskRunResult 

type TaskRunResult struct {
	// TaskResultCommon embeds most of result properties.
	TaskResultCommon

	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the task, see TaskRunResultKey().
	Key *datastore.Key `gae:"$key"`

	// RequestCreated is copied from the corresponding TaskRequest entity.
	RequestCreated time.Time `gae:"request_created,noindex"`

	// RequestTags are copied from the corresponding TaskRequest entity.
	RequestTags []string `gae:"request_tags,noindex"`

	// RequestName is copied from the corresponding TaskRequest entity.
	RequestName string `gae:"request_name,noindex"`

	// RequestUser is copied from the corresponding TaskRequest entity.
	RequestUser string `gae:"request_user,noindex"`

	// BotID is ID of a bot that runs this task attempt.
	//
	// The index is used in task listing queries to filter by a specific bot.
	BotID string `gae:"bot_id"`

	// CostUSD is an approximate bot time cost spent executing this task.
	CostUSD float64 `gae:"cost_usd,noindex"`

	// Killing is true if a user requested the task to be canceled while it is
	// already running.
	//
	// Setting this field eventually results in the task moving to KILLED state.
	// This transition also unsets Killing back to false.
	Killing bool `gae:"killing,noindex"`

	// DeadAfter specifies the time after which the bot executing this task
	// is considered dead.
	//
	// It is set after every ping from the bot if the task is in RUNNING state
	// and get unset once the task terminates.
	DeadAfter datastore.Optional[time.Time, datastore.Unindexed] `gae:"dead_after_ts"`
}

TaskRunResult contains result of an attempt to run a task on a bot.

Parent is a TaskResultSummary. Key id is 1.

Unlike TaskResultSummary it is created only when the task is assigned to a bot, i.e. existence of this entity means a bot claimed the task and started executing it.

func (*TaskRunResult) PerformanceStatsKey 

func (p *TaskRunResult) PerformanceStatsKey(ctx context.Context) *datastore.Key

PerformanceStatsKey returns PerformanceStats entity key or nil.

It returns a non-nil key if the performance stats entity can **potentially** exist (e.g. the task has finished running and reported its stats). It returns nil if the performance stats entity definitely doesn't exist.

func (*TaskRunResult) TaskRequestKey 

func (p *TaskRunResult) TaskRequestKey() *datastore.Key

TaskRequestKey returns the parent task request key or panics if it is unset.

func (*TaskRunResult) ToProto 

func (p *TaskRunResult) ToProto() *apipb.TaskResultResponse

ToProto converts the TaskRunResult struct to an apipb.TaskResultResponse.

Note: This function will not handle PerformanceStats due to the requirement of fetching another datastore entity. Please refer to PerformanceStats to fetch them.

TODO(2026): Older entities do not have some fields populated (in particular Name). The caller will have to fetch them separately from the corresponding TaskRequest entity. See TaskRunResultMissingFieldsFromRequest. This fallback can be removed in ~2026.

type TaskSlice 

type TaskSlice struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Properties defines where and how to run the task.
	//
	// If a task is marked as an idempotent (see TaskProperties), property values
	// are hashed in a reproducible way and the final hash is used to find a
	// previously succeeded task that has the same properties.
	Properties TaskProperties `gae:"properties,lsp"`

	// PropertiesHash is a precalculated hash of properties.
	//
	// Populated when the task is scheduled. It is used for task deduplication and
	// in BQ exports.
	PropertiesHash []byte `gae:"properties_hash,noindex"`

	// ExpirationSecs defines how long the slice can sit in a pending queue.
	//
	// If this task slice is not scheduled by this moment, the next one will be
	// enqueued instead.
	ExpirationSecs int64 `gae:"expiration_secs"`

	// WaitForCapacity is a legacy flag that does nothing, always false now.
	//
	// TODO(vadimsh): Remove it when no longer referenced
	WaitForCapacity bool `gae:"wait_for_capacity"`
}

TaskSlice defines where and how to run the task and when to give up.

The task will fallback from one slice to the next until it finds a matching bot.

This entity is not saved in the DB as a standalone entity, instead it is embedded in a TaskRequest, unindexed.

This entity is immutable.

func (*TaskSlice) ToProto 

func (p *TaskSlice) ToProto() *apipb.TaskSlice

ToProto returns an apipb.TaskSlice version of the TaskSlice.

type TaskToRun 

type TaskToRun struct {
	// Extra are entity properties that didn't match any declared ones below.
	//
	// Should normally be empty.
	Extra datastore.PropertyMap `gae:"-,extra"`

	// Key identifies the task and its slice, see TaskToRunKey().
	//
	// Note that the kind is TaskToRunShard<index>, see TaskToRunKind().
	Key *datastore.Key `gae:"$key"`

	// Created is used to know when the entity is enqueued.
	//
	// The very first TaskToRun has the same value as TaskRequest.Created, but the
	// following ones (when using multiple task slices) have Created set at the
	// time they are created.
	//
	// Used in both native and RBE mode.
	Created time.Time `gae:"created_ts,noindex"`

	// Dimensions is a copy of dimensions from the corresponding task slice of
	// TaskRequest.
	//
	// It is used to quickly check if a bot can reap this TaskToRun right after
	// fetching it from a datastore query.
	//
	// Used in both native and RBE mode.
	Dimensions TaskDimensions `gae:"dimensions"`

	// RBEReservation is the RBE reservation name that is (or will be) handling
	// this TaskToRun.
	//
	// If set, then TaskToRunShard is in RBE mode. If not, then in native
	// mode. TaskToRunShard in RBE mode are always (transactionally) created with
	// a Task Queue task to actually dispatch them to the RBE scheduler.
	RBEReservation string `gae:"rbe_reservation,noindex"`

	// Expiration is the scheduling deadline for this TaskToRun.
	//
	// It is based on TaskSlice.Expiration. It is used to figure out when to
	// fallback on the next task slice. It is scanned by a cron job and thus needs
	// to be indexed.
	//
	// It is unset when the TaskToRun is claimed, canceled or expires.
	//
	// Used in both native and RBE mode.
	Expiration datastore.Optional[time.Time, datastore.Indexed] `gae:"expiration_ts"`

	// QueueNumber is a magical number by which bots and tasks find one another.
	//
	// Used only in native mode. Always unset and unused in RBE mode.
	//
	// Priority and request creation timestamp are mixed together to allow queries
	// to order the results by this field to allow sorting by priority first, and
	// then timestamp.
	//
	// Gets unset when the TaskToRun is consumed.
	QueueNumber datastore.Optional[int64, datastore.Indexed] `gae:"queue_number"`

	// ClaimID is set if some bot claimed this TaskToRun and will execute it.
	//
	// Used only in RBE mode. Always unset in native mode.
	//
	// It is an opaque ID supplied by the bot when it attempts to claim this
	// entity. If TaskToRun is already claimed and ClaimID matches the one
	// supplied by the bot, then it means this bot has actually claimed the entity
	// already and now just retries the call.
	//
	// Never gets unset once set.
	ClaimID datastore.Optional[string, datastore.Unindexed] `gae:"claim_id"`

	// ExpirationDelay is a delay from Expiration to the actual expiry time.
	//
	// This is set at expiration process if the last task slice expired by
	// reaching its deadline. Unset if the last slice expired because there were
	// no bots that could run it.
	//
	// Exclusively for monitoring.
	ExpirationDelay datastore.Optional[float64, datastore.Unindexed] `gae:"expiration_delay"`
}

TaskToRun defines a TaskRequest slice ready to be scheduled on a bot.

Each TaskRequest results in one or more TaskToRun entities (one per slice). They are created sequentially, one by one, as the task progresses through its slices. Each TaskToRun is eventually either picked up by a bot for execution or expires. Each TaskToRun picked up for execution has an TaskRunResult entity (expired ones don't).

A TaskToRun can either be in "native mode" (dispatched via the native Swarming scheduler implemented in Python code base) or in "RBE mode" (dispatched via the remote RBE scheduler service). This is controlled by RBEReservation field.

A TaskToRun (regardless of mode) can be in two states:

1. "reapable"

  • Native mode: QueueNumber and Expiration are both set.
  • RBE mode: ClaimID is unset and Expiration is set.

2. "consumed":

  • Native mode: QueueNumber and Expiration are both unset.
  • RBE mode: ClaimID is set and Expiration is unset.

The entity starts its life in reapable state and then transitions to consumed state either by being picked up by a bot for execution or when it expires. Consumed state is final.

The key ID is (see TaskToRunID): - lower 4 bits is the try number. The only supported value is 1 now. - next 5 bits are TaskResultSummary.CurrentTaskSlice (shifted by 4 bits). - the rest is 0.

This entity is stored using a bunch of different shards. The shard number is derived deterministically by calculating dimensions hash % TaskToRunShards, see TaskToRunKey.

func (*TaskToRun) Consume 

func (t *TaskToRun) Consume(claimID string)

Consume moves t into non-reapable state (e.g. when canceling).

func (*TaskToRun) IsReapable 

func (t *TaskToRun) IsReapable() bool

IsReapable returns true if the TaskToRun is still pending.

Source Files

View all Source files

Directories

Path Synopsis
Package internalmodelpb contains messages used by Swarming data model.
 Package internalmodelpb contains messages used by Swarming data model.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.