builder

package
v0.0.0-...-d03d5e3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 15, 2024 License: Apache-2.0 Imports: 55 Imported by: 2

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConvertCommandToShellScript 

func ConvertCommandToShellScript(command *remoteexecution.Command, w io.StringWriter) error

ConvertCommandToShellScript writes a POSIX shell script to a StringWriter that causes a process to be launched in the way encoded in a Command message.

Because input roots do not explicitly store parent directories of outputs, and actions generally assume that they exist, the resulting shell script may contain one or more "mkdir -p" calls to create those directories prior to execution.

func GetResultAndGRPCCodeFromExecuteResponse 

func GetResultAndGRPCCodeFromExecuteResponse(response *remoteexecution.ExecuteResponse) (result, grpcCode string)

GetResultAndGRPCCodeFromExecuteResponse converts an ExecuteResponse to a pair of strings that describe the execution outcome. These strings can be used as part of metrics labels.

TODO: Move this into some other package, so that pkg/scheduler doesn't need to depend on pkg/builder?

func LaunchWorkerThread 

func LaunchWorkerThread(group program.Group, buildClient *BuildClient, workerName string)

LaunchWorkerThread launches a single routine that uses a build client to repeatedly synchronizes against the scheduler, requesting a task to execute.

func NewDefaultExecuteResponse 

func NewDefaultExecuteResponse(request *remoteworker.DesiredState_Executing) *remoteexecution.ExecuteResponse

NewDefaultExecuteResponse creates an ExecuteResponse message that contains all fields that BuildExecutor should set by default.

Types

type BuildClient 

type BuildClient struct {
	// contains filtered or unexported fields
}

BuildClient is a client for the Remote Worker protocol. It can send synchronization requests to a scheduler, informing it of the current state of the worker, while also obtaining requests for executing build actions.

func NewBuildClient 

func NewBuildClient(scheduler remoteworker.OperationQueueClient, buildExecutor BuildExecutor, filePool filesystem.FilePool, clock clock.Clock, workerID map[string]string, instanceNamePrefix digest.InstanceName, platform *remoteexecution.Platform, sizeClass uint32) *BuildClient

NewBuildClient creates a new BuildClient instance that is set to the initial state (i.e., being idle).

func (*BuildClient) Run 

func (bc *BuildClient) Run(ctx context.Context) (bool, error)

Run an iteration of the Remote Worker client, by performing a single synchronization against the scheduler.

type BuildDirectory 

type BuildDirectory interface {
	ParentPopulatableDirectory
	UploadableDirectory

	// Methods inherited from filesystem.Directory.
	Mknod(name path.Component, perm os.FileMode, deviceNumber filesystem.DeviceNumber) error
	Remove(name path.Component) error
	RemoveAll(name path.Component) error

	// Identical to EnterDirectory(), except that it returns a
	// BuildDirectory object.
	EnterBuildDirectory(name path.Component) (BuildDirectory, error)

	// Installs a set of hooks into the directory to intercept I/O
	// operations. The FilePool may be used to allocate storage
	// space. The ErrorLogger may be used to report fatal I/O
	// errors. Implementations of BuildDirectory are free to let
	// this be a no-op, with the disadvantage that they cannot apply
	// resource limits or provide rich I/O error messages.
	InstallHooks(filePool re_filesystem.FilePool, errorLogger util.ErrorLogger)

	// Recursively merges the contents of a Directory stored in the
	// Content Addressable Storage into a local directory. If this
	// process is synchronous, this function can return a
	// synchronous error. If this process is lazy/asynchronous, the
	// provided ErrorLogger may be used to return an error.
	MergeDirectoryContents(ctx context.Context, errorLogger util.ErrorLogger, digest digest.Digest, monitor access.UnreadDirectoryMonitor) error
}

BuildDirectory is a directory that may be used for the purpose of running build actions. BuildDirectory shares some operations with filesystem.Directory, but it has a couple of custom operations that implementations may use to run actions in a more efficient and manageable way.

func NewNaiveBuildDirectory 

func NewNaiveBuildDirectory(directory filesystem.DirectoryCloser, directoryFetcher cas.DirectoryFetcher, fileFetcher cas.FileFetcher, fileFetcherSemaphore *semaphore.Weighted, contentAddressableStorage blobstore.BlobAccess) BuildDirectory

NewNaiveBuildDirectory creates a BuildDirectory that is backed by a simple filesystem.Directory with all of the operations implemented in a naive way. Namely, MergeDirectoryContents() recursively loads all directories from the Content Addressable Storage (CAS) and requests that all of their files are copied into the build directory.

This implementation is intended to be used in combination with regular local file systems. The downside of such file systems is that we cannot populate them on demand. All of the input files must be present before invoking the build action.

func NewVirtualBuildDirectory 

func NewVirtualBuildDirectory(directory virtual.PrepopulatedDirectory, directoryFetcher cas.DirectoryFetcher, contentAddressableStorage blobstore.BlobAccess, symlinkFactory virtual.SymlinkFactory, characterDeviceFactory virtual.CharacterDeviceFactory, handleAllocator virtual.StatefulHandleAllocator) BuildDirectory

NewVirtualBuildDirectory creates a BuildDirectory that is backed by a virtual.PrepopulatedDirectory. Instead of creating all files in the input root explicitly, it calls PrepopulatedDirectory.CreateChildren to add special file and directory nodes whose contents are read on demand.

type BuildDirectoryCreator 

type BuildDirectoryCreator interface {
	GetBuildDirectory(ctx context.Context, actionDigestIfNotRunInParallel *digest.Digest) (BuildDirectory, *path.Trace, error)
}

BuildDirectoryCreator is used by LocalBuildExecutor to obtain build directories in which build actions are executed.

func NewCleanBuildDirectoryCreator 

func NewCleanBuildDirectoryCreator(base BuildDirectoryCreator, idleInvoker *cleaner.IdleInvoker) BuildDirectoryCreator

NewCleanBuildDirectoryCreator is an adapter for BuildDirectoryCreator that upon acquisition and release calls into a Cleaner. This Cleaner may, for example, be set up to empty out the build directory. This guarantees that build actions aren't able to see data left behind by ones that ran previously.

func NewRootBuildDirectoryCreator 

func NewRootBuildDirectoryCreator(buildDirectory BuildDirectory) BuildDirectoryCreator

NewRootBuildDirectoryCreator is a BuildDirectoryCreator that repeatedly hands out a single directory present on the current system. Additional decorators are used to run builds in subdirectories, so that build actions may run in parallel.

func NewSharedBuildDirectoryCreator 

func NewSharedBuildDirectoryCreator(base BuildDirectoryCreator, nextParallelActionID *atomic.Uint64) BuildDirectoryCreator

NewSharedBuildDirectoryCreator is an adapter for BuildDirectoryCreator that causes build actions to be executed inside a subdirectory within the build directory, as opposed to inside the build directory itself. The subdirectory is either named after the action digest of the build action or uses an incrementing number, based on whether collisions may occur.

This adapter can be used to add concurrency to a single worker. When executing build actions in parallel, every build action needs its own build directory.

type BuildExecutor 

type BuildExecutor interface {
	CheckReadiness(ctx context.Context) error
	Execute(ctx context.Context, filePool filesystem.FilePool, monitor access.UnreadDirectoryMonitor, digestFunction digest.Function, request *remoteworker.DesiredState_Executing, executionStateUpdates chan<- *remoteworker.CurrentState_Executing) *remoteexecution.ExecuteResponse
}

BuildExecutor is the interface for the ability to run Bazel execute requests and yield an execute response.

func NewCachingBuildExecutor 

func NewCachingBuildExecutor(base BuildExecutor, contentAddressableStorage, actionCache blobstore.BlobAccess, browserURL *url.URL) BuildExecutor

NewCachingBuildExecutor creates an adapter for BuildExecutor that stores action results in the Action Cache (AC) if they may be cached. If they may not be cached, they are stored in the Content Addressable Storage (CAS) instead.

In both cases, a link to bb_browser is added to the ExecuteResponse, so that the user may inspect the Action and ActionResult in detail.

func NewCompletedActionLoggingBuildExecutor 

func NewCompletedActionLoggingBuildExecutor(base BuildExecutor, uuidGenerator util.UUIDGenerator, logger CompletedActionLogger, instanceNamePatcher digest.InstanceNamePatcher) BuildExecutor

NewCompletedActionLoggingBuildExecutor returns a new completedActionLoggingBuildExecutor that will transmit CompletedActions to an external server for real-time analysis of REv2 Action metadata using a CompletedActionLogger.

func NewCostComputingBuildExecutor 

func NewCostComputingBuildExecutor(base BuildExecutor, expensesPerSecond map[string]*resourceusage.MonetaryResourceUsage_Expense) BuildExecutor

NewCostComputingBuildExecutor wraps an existing BuildExecutor, adding the computed cost of the action to the prepopulated AuxiliaryMetadata field of the ActionResult. The provided expenses are represented on a per-second basis and are then multiplied by the amount of seconds that it took for a worker to complete the Action.

func NewFilePoolStatsBuildExecutor 

func NewFilePoolStatsBuildExecutor(buildExecutor BuildExecutor) BuildExecutor

NewFilePoolStatsBuildExecutor creates a decorator for BuildExecutor that annotates ExecuteResponses to contain usage statistics of the FilePool. FilePools are used to allocate temporary files that are generated by the build action (e.g., output files).

func NewLocalBuildExecutor 

func NewLocalBuildExecutor(contentAddressableStorage blobstore.BlobAccess, buildDirectoryCreator BuildDirectoryCreator, runner runner_pb.RunnerClient, clock clock.Clock, maximumWritableFileUploadDelay time.Duration, inputRootCharacterDevices map[path.Component]filesystem.DeviceNumber, maximumMessageSizeBytes int, environmentVariables map[string]string, forceUploadTreesAndDirectories bool) BuildExecutor

NewLocalBuildExecutor returns a BuildExecutor that executes build steps on the local system.

func NewLoggingBuildExecutor 

func NewLoggingBuildExecutor(base BuildExecutor, browserURL *url.URL) BuildExecutor

NewLoggingBuildExecutor wraps an existing BuildExecutor, adding basic logging. A link to bb_browser is printed prior to executing the action. A JSON representation of the ExecuteResponse is logged after completion.

func NewMetricsBuildExecutor 

func NewMetricsBuildExecutor(buildExecutor BuildExecutor) BuildExecutor

NewMetricsBuildExecutor creates a decorator for BuildExecutor that exposes the statistics stored in ExecutedActionMetadata as Prometheus metrics.

func NewNoopBuildExecutor 

func NewNoopBuildExecutor(contentAddressableStorage blobstore.BlobAccess, maximumMessageSizeBytes int, browserURL *url.URL) BuildExecutor

NewNoopBuildExecutor creates a BuildExecutor that always returns an error message when attempting to execute an action.

This implementation may be used to force a build client to upload the input root of an action into the Content Addressable Storage (CAS) without causing it to be executed afterwards. This may be useful when attempting to debug actions.

func NewPrefetchingBuildExecutor 

func NewPrefetchingBuildExecutor(buildExecutor BuildExecutor, contentAddressableStorage blobstore.BlobAccess, directoryFetcher cas.DirectoryFetcher, fileReadSemaphore *semaphore.Weighted, fileSystemAccessCache blobstore.BlobAccess, maximumMessageSizeBytes, bloomFilterBitsPerElement, bloomFilterMaximumSizeBytes int) BuildExecutor

NewPrefetchingBuildExecutor creates a decorator for BuildExecutor that as the action gets executed, prefetches files that are part of the action's input root from the Content Addressable Storage (CAS). It determines which files to download by making use of Bloom filters stored in the File System Access Cache (FSAC).

It also monitors the file system access of the action to be executed. If this yields a Bloom filter that differs from the one retried from the FSAC, it stores an updated version. This ensures that the next time a similar action is ran, only the files that are expected to be used are downloaded.

This decorator is only of use on workers that use a virtual build directory (FUSE, NFSv4). On workers that use native build directories, the monitor is ignored, leading to empty Bloom filters being stored.

func NewStorageFlushingBuildExecutor 

func NewStorageFlushingBuildExecutor(base BuildExecutor, flush StorageFlusher) BuildExecutor

NewStorageFlushingBuildExecutor is an adapter for BuildExecutor that calls a callback after every operation. The callback is typically used to flush pending writes to underlying storage, to ensure that other processes in the cluster have a consistent view of the completion of the operation.

func NewTestInfrastructureFailureDetectingBuildExecutor 

func NewTestInfrastructureFailureDetectingBuildExecutor(base BuildExecutor, shutdownState *TestInfrastructureFailureShutdownState, maximumConsecutiveFailures uint32) BuildExecutor

NewTestInfrastructureFailureDetectingBuildExecutor is a decorator for BuildExecutor that counts the number of consecutive actions that generated one or more "test.infrastructure_failure" output files. If the count exceeds a configured value, the BuildExecutor will start to fail readiness checks. This prevents further work from being executed.

This decorator is useful when workers have peripherals attached to them that are prone to hardware failures. Bazel allows tests to report these failures by creating the file designated by the TEST_INFRASTRUCTURE_FAILURE_FILE environment variable.

Please refer to the Bazel test encyclopedia for more details on TEST_INFRASTRUCTURE_FAILURE_FILE: https://bazel.build/reference/test-encyclopedia

func NewTimestampedBuildExecutor 

func NewTimestampedBuildExecutor(buildExecutor BuildExecutor, clock clock.Clock, workerName string) BuildExecutor

NewTimestampedBuildExecutor creates a decorator for BuildExecutor that augments the ActionResult that is part of the ExecuteResponse with ExecutedActionMetadata. More concretely, it ensures that the ActionResult contains the name of the worker performing the build and timing information.

func NewTracingBuildExecutor 

func NewTracingBuildExecutor(buildExecutor BuildExecutor, tracerProvider trace.TracerProvider) BuildExecutor

NewTracingBuildExecutor is a decorator for BuildExecutor that creates an OpenTelemetry trace span for every action that is executed. At the start of every execution state, an event is added to the span that indicates which state is entered.

type CompletedActionLogger 

type CompletedActionLogger interface {
	LogCompletedAction(completedAction *cal_proto.CompletedAction)
}

The CompletedActionLogger can be used to record CompletedActions for realtime or post-build analysis in a remote service. This is particularly useful for understanding how build actions change over time by inspecting the aggregated CompletedAction metadata.

type OutputHierarchy 

type OutputHierarchy struct {
	// contains filtered or unexported fields
}

OutputHierarchy is used by LocalBuildExecutor to track output directories and files that are expected to be generated by the build action. OutputHierarchy can be used to create parent directories of outputs prior to execution, and to upload outputs into the CAS after execution.

func NewOutputHierarchy 

func NewOutputHierarchy(command *remoteexecution.Command) (*OutputHierarchy, error)

NewOutputHierarchy creates a new OutputHierarchy that uses the working directory and the output paths specified in an REv2 Command message.

func (*OutputHierarchy) CreateParentDirectories 

func (oh *OutputHierarchy) CreateParentDirectories(d ParentPopulatableDirectory) error

CreateParentDirectories creates parent directories of outputs. This function is called prior to executing the build action.

func (*OutputHierarchy) UploadOutputs 

func (oh *OutputHierarchy) UploadOutputs(ctx context.Context, d UploadableDirectory, contentAddressableStorage blobstore.BlobAccess, digestFunction digest.Function, writableFileUploadDelay <-chan struct{}, actionResult *remoteexecution.ActionResult, forceUploadTreesAndDirectories bool) error

UploadOutputs uploads outputs of the build action into the CAS. This function is called after executing the build action.

type ParentPopulatableDirectory 

type ParentPopulatableDirectory interface {
	Close() error
	EnterParentPopulatableDirectory(name path.Component) (ParentPopulatableDirectory, error)
	Mkdir(name path.Component, perm os.FileMode) error
}

ParentPopulatableDirectory contains a subset of the methods of filesystem.Directory that are required for creating the parent directories of output files of a build action.

type RemoteCompletedActionLogger 

type RemoteCompletedActionLogger struct {
	// contains filtered or unexported fields
}

The RemoteCompletedActionLogger type is used to store and send CompletedActions for a completedActionLoggingBuildExecutor. It keeps track of which messages have been previously transmitted and retries them if our connection with the server is interrupted.

func NewRemoteCompletedActionLogger 

func NewRemoteCompletedActionLogger(queueSize int, client cal_proto.CompletedActionLoggerClient) *RemoteCompletedActionLogger

NewRemoteCompletedActionLogger returns a new RemoteCompletedActionLogger with a predefined maximum capacity of stored messages. This ensures that we don't overwhelm the server in case it is under heavy load and cannot respond.

func (*RemoteCompletedActionLogger) LogCompletedAction 

func (logger *RemoteCompletedActionLogger) LogCompletedAction(completedAction *cal_proto.CompletedAction)

LogCompletedAction will add one CompletedAction to the RemoteCompletedActionLogger and notify that a message is ready to be transmitted.

func (*RemoteCompletedActionLogger) SendAllCompletedActions 

func (logger *RemoteCompletedActionLogger) SendAllCompletedActions() error

SendAllCompletedActions is responsible for managing goroutines that perform the rpc transmission and response handling.

type StorageFlusher 

type StorageFlusher func(context.Context) error

StorageFlusher is a callback that is invoked by NewStorageFlushingBuildExecutor to flush contents to storage.

type TestInfrastructureFailureShutdownState 

type TestInfrastructureFailureShutdownState struct {
	// contains filtered or unexported fields
}

TestInfrastructureFailureShutdownState keeps track of whether a group of worker threads have shut down, due to an excessive number of consecutive tests failing due to infrastructure failures on a single worker thread.

func NewTestInfrastructureFailureShutdownState 

func NewTestInfrastructureFailureShutdownState() *TestInfrastructureFailureShutdownState

NewTestInfrastructureFailureShutdownState creates a new TestInfrastructureFailureShutdownState that is in the initial state, where no infrastructure failures have occurred.

type UploadableDirectory 

type UploadableDirectory interface {
	// Methods inherited from filesystem.Directory.
	Close() error
	EnterUploadableDirectory(name path.Component) (UploadableDirectory, error)
	Lstat(name path.Component) (filesystem.FileInfo, error)
	ReadDir() ([]filesystem.FileInfo, error)
	Readlink(name path.Component) (path.Parser, error)

	// Upload a file into the Content Addressable Storage.
	//
	// Implementations that are capable of detecting that files
	// remain opened for writing may block until they are closed. To
	// ensure this does not end up blocking indefinitely, a channel
	// is provided that gets closed after a configured amount of
	// time. This channel is also closed when the Context is done.
	UploadFile(ctx context.Context, name path.Component, digestFunction digest.Function, writableFileUploadDelay <-chan struct{}) (digest.Digest, error)
}

UploadableDirectory is a directory that can be uploaded into the Content Addressable Storage. It is provided to OutputHierarchy.UploadOutputs(), which traverses it and uploads paths that were specified in the Action message.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.