Documentation
¶
Overview ¶
Package cfg implements Swarming server configuration logic.
It consists of 3 components:
- A cron job that periodically fetches all server configs at the most recent revision, preprocesses them and stores the result in the local datastore in a compact form.
- A background goroutine running in every server process that periodically polls configs stored in the datastore, unpacks and transforms them into a form optimized for querying from RPC handlers. Its job is to maintain in the local process memory the most recent config in its queryable form.
- A validation logic used by LUCI Config callback to validate configs in config presubmit checks and used by the cron job to double check new configs are good.
This package is also responsible for assembling the deployable bot package based on the server configuration and the bot package in CIPD.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func HostBotID ¶
HostBotID takes a bot ID like `<host>--<sfx>` and returns just `<host>`.
Bot IDs like `<host>--<sfx>` are called composite. They are used to represent multiple bots running on the same host (e.g. as docker containers) sharing the same host credentials. The `<host>` part identifies this host. It is used when checking the authentication tokens and looking up the bot group config.
If the bot ID is not composite, returns it as is.
func UpdateConfigs ¶
func UpdateConfigs(ctx context.Context, ebs *EmbeddedBotSettings, cipdClient CIPD) error
UpdateConfigs fetches the most recent server configs and bot code and stores them in the local datastore if they appear to be valid.
If `cipdClient` is not nil, uses it to communicate with CIPD. Otherwise constructs a default client.
Called from a cron job once a minute.
Types ¶
type BotArchiveInfo ¶
type BotArchiveInfo struct {
// Digest is the bot archive SHA256 digest aka "bot archive version".
Digest string `gae:",noindex"`
// Chunks is the list of BotArchiveChunk entities with the archive content.
Chunks []string `gae:",noindex"`
// BotConfigHash is SHA256 of the bot_config.py embedded into the bot archive.
BotConfigHash string `gae:",noindex"`
// BotConfigRev is the revision of bot_config.py script used.
BotConfigRev string `gae:",noindex"`
// PackageInstanceID is the CIPD instance ID of the base bot package.
PackageInstanceID string `gae:",noindex"`
// PackageServer is an URL of the CIPD server with the bot package.
PackageServer string `gae:",noindex"`
// PackageName is the CIPD package name e.g. "luci/swarming/swarming_bot".
PackageName string `gae:",noindex"`
// PackageVersion is e.g. "git_commit:..." CIPD tag or ref.
PackageVersion string `gae:",noindex"`
}
BotArchiveInfo contains information about a concrete assembled bot archive.
It contains both some inputs to the bot archive building process and its output: the bot archive digest and references to its chunks in the datastore that can be assembled together to get the final zip file.
Inputs are available through the settings.cfg config as well, but they are copied here for convenience when logging changes.
func (*BotArchiveInfo) FetchBotArchive ¶
func (ba *BotArchiveInfo) FetchBotArchive(ctx context.Context) ([]byte, error)
FetchBotArchive fetches the bot archive blob from the datastore.
type BotGroup ¶
type BotGroup struct {
// Dimensions is bot dimensions assigned to matched bots via the config.
//
// Includes as least "pool" dimension, but potentially more.
Dimensions map[string][]string
// Auth defines how to authenticate bot API calls.
//
// There's always at least one element (but can be more if multiple auth
// methods are allowed).
Auth []*configpb.BotAuth
// SystemServiceAccount is what account to use on bots when authenticating
// calls to various system-level services (like CAS and CIPD) required for
// correct operation of the bot.
//
// Either an empty string, a literal string "bot" or a service account email.
SystemServiceAccount string
// BotConfigScript is an optional name of a custom hooks script.
//
// Its existence is validated when ingesting the config.
BotConfigScript string
}
BotGroup is one parsed section of bots.cfg config.
It defines configuration that applies to all bots within that section.
type CIPD ¶
type CIPD interface {
// ResolveVersion resolves a version label into a CIPD instance ID.
ResolveVersion(ctx context.Context, server, cipdpkg, version string) (string, error)
// FetchInstance fetches contents of a package given via its instance ID.
FetchInstance(ctx context.Context, server, cipdpkg, iid string) (pkg.Instance, error)
}
CIPD is used to communicate with the CIPD server.
Usually it is *cipd.Client.
type Config ¶
type Config struct {
// VersionInfo contains versions of the fetched configs.
VersionInfo VersionInfo
// Refreshed is the local time when the config was fetched from the datastore.
Refreshed time.Time
// contains filtered or unexported fields
}
Config is an immutable queryable representation of Swarming server configs.
It is a snapshot of configs at some particular revision. Use an instance of Provider to get it.
func (*Config) BotGroup ¶
BotGroup returns a BotGroup config matching the given bot ID.
Understands composite bot IDs, see HostBotID(...). Always returns some config (never nil). If there's no config assigned to a bot, returns a default config.
func (*Config) RouteToGoPercent ¶
RouteToGoPercent returns how much traffic to this route should be handled by the Go server (vs Python server).
Returns a number in range [0; 100].
func (*Config) Settings ¶
func (cfg *Config) Settings() *configpb.SettingsCfg
Settings are settings proto with defaults filled in.
type EmbeddedBotSettings ¶
type EmbeddedBotSettings struct {
// ServerURL is "https://..." URL of the Swarming server itself.
ServerURL string
}
EmbeddedBotSettings is configuration data that eventually ends up in the bot zip archive inside config.json.
type Pool ¶
type Pool struct {
// Realm is a realm with ACLs for this pool's resources.
//
// This is a global realm name, e.g. `infra:pool/flex/try`.
Realm string
// DefaultTaskRealm is a realm for tasks if they don't have a realm set.
//
// It is optional. If not set, all tasks must have the realm specified by
// the caller when they are created.
DefaultTaskRealm string
}
Pool is a parsed config of some single pool.
type Provider ¶
type Provider struct {
// contains filtered or unexported fields
}
Provider knows how to load preprocessed configs from the datastore and convert them into instances of Config.
Provider is a long-living object that keeps a reference to the most recent config, periodically reloading it from the datastore.
func NewProvider ¶
NewProvider initializes a Provider by fetching the initial copy of configs.
If there are no configs stored in the datastore (happens when bootstrapping a new service), uses some default empty config.
func (*Provider) Cached ¶
Cached returns an immutable snapshot of the currently cached config.
Returns the config cached in the local process memory. It may be slightly behind the most recently ingested config. Use Latest() to bypass the cache and get the most recently ingested config instead.
Note that multiple sequential calls to Cached() may return different snapshots if the config changes between them. For that reason it is better to get the snapshot once at the beginning of an RPC handler, and use it throughout.
func (*Provider) FreshEnough ¶
FreshEnough returns an immutable snapshot of a config either by using the local process cache (if it is fresh enough) or by fetching the latest config from the datastore.
`seen` should be a VersionInfo.Fetched value from some previously seen config version.
If the VersionInfo.Fetched of the currently cached config is equal or larger than `seen`, returns the cached config. Otherwise fetches the latest config (its VersionInfo.Fetched will be larger or equal than `seen`).
This is used to avoid going back to a previous config version once a process seen a newer config version at least once.
func (*Provider) Latest ¶
Latest returns an immutable snapshot of the most recently ingested config by fetching it from the datastore.
Can be used in places where staleness of the cached config is unacceptable.
Updates the local cache as a side effect.
func (*Provider) RefreshPeriodically ¶
RefreshPeriodically runs a loop that periodically refetches the config from the datastore to update the local cache.
type VersionInfo ¶
type VersionInfo struct {
// Digest is deterministically derived from the fully expanded configs
// (including the bot code archive fetched from CIPD).
//
// It is used by the config cache to know if something has really changed
// and the in-memory copy of the config needs to be reloaded.
Digest string `gae:",noindex"`
// Fetched is when this config content was fetched for the first time.
//
// This is the first time this specific config digest was seen. Always
// monotonically increases. Has milliseconds precision.
Fetched time.Time `gae:",noindex"`
// Revision is the config repo git commit processed most recently.
//
// Note that Revision may changed even if Digest stays the same (happens if
// some config file we don't use changes).
//
// This is FYI mostly. Digest is used in all important places to figure out
// if config really changes.
Revision string `gae:",noindex"`
// Touched is when this config was last touched by the UpdateConfig cron.
//
// This is FYI mostly. It is updated whenever a new revision is seen, even if
// it doesn't change the digest. Has milliseconds precision.
Touched time.Time `gae:",noindex"`
// StableBot is information about the current stable bot archive version.
//
// Derived based on the BotDeployment config section and the state in CIPD.
StableBot BotArchiveInfo `gae:",noindex"`
// CanaryBot is information about the current canary bot archive version.
//
// Derived based on the BotDeployment config section and the state in CIPD.
CanaryBot BotArchiveInfo `gae:",noindex"`
}
VersionInfo identifies versions of Swarming configs.
It is stored in the datastore as part of ConfigBundle and ConfigBundleRev.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cfgtest allows to mock Swarming configs for tests.
|
Package cfgtest allows to mock Swarming configs for tests. |
|
Package internalcfgpb contains messages used by Swarming config system.
|
Package internalcfgpb contains messages used by Swarming config system. |