bazel

README

Bazel Remote Execution for Scoot

This contains initial server support for the Bazel gRPC Remote Execution API. This consists of two distinct concepts, the Execution API and ContentAddressableStore (CAS) API.

The Bazel API Design Document can be found at: https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s/edit#

Components:

• cas/ contains CAS API server implementation
• execution/ contains Execution API server implementation
• ./ (bazel) contains general Bazel constants, utils, and a gRPC server abstraction

Running/testing the API:

• Scheduler will initialize and serve the Execution API over gRPC on the default port.
• Apiserver will initialize and serve the CAS API over gRPC on the default port.

go install github.com/twitter/scoot/binaries/scheduler github.com/twitter/scoot/binaries/apiserver
./scheduler
./apiserver

BZUtil CLI Client

The preferred client for Scoot operations is binaries/bzutil/main.go, which implements GRPC client interfaces. For more raw testing of service interfaces, the generic grpc_cli client can be used.

FS_UTIL Client

We leverage the fs_util client binary which is managed by the https://github.com/pantsbuild/pants/ project. We coordinate to find suitable releases and specify them in the get_fs_util.sh script. The project Makefile will fetch the correct client and make it available where needed by Scoot. Binaries are fetched from https://binaries.pantsbuild.org

GRPC CLI Client

Testing the gRPC APIs can be done with the grpc_cli tool, but formatting the proper data structures is difficult, so this method is unsupported. This has to be built and installed from the grpc repo. See: https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md

Twitter MacBook installation pointers

In general, the grpc install instructions are accurate, but dependency installation via brew might not work. Use a global, non-MDE copy of brew to install dependencies such as gflags, as MDE brew install will not make these libraries accessible when trying to compile and link the grpc binaries.

Usage Example

Remote Execution End-to-End Test Example with BZUtil

This details how to run a Bazel Remote Execution request in Scoot from scratch. The worflow will be:

1. Start up a scheduler, apiserver, and workerserver
2. Upload the request Command (argv and env) to the apiserver via the CAS API
3. Upload the request input directory to the apiserver via fs_util/CAS API
4. Upload the request Action (references Command and input) to the apiserver via the CAS API
5. Schedule the request on the scheduler via the Execution API
6. Get results of the request from the scheduler via the Longrunning API

1:

$GOPATH/bin/setup-cloud-scoot --strategy local.local

2:

bzutil upload_command --cas_addr="localhost:12100" sleep 17
INFO[0000] Using argv: ["sleep" "17"] env: map[]         sourceLine="bzutil/main.go:112"
INFO[0000] Wrote to CAS successfully                     sourceLine="bzutil/main.go:142"
1833d7c57656d2b7ee97e2068ce742f80e61357fba12a8b8d627782da3a58c29/11

3: Example uploading contents of ~/workspace/bazel/dir/. Note that ~/workspace/bazel/db/ must exist, but can be empty. We don't need this dir for our particular command, but we're uploading one as an example anyway:

fs_util --local-store-path=/Users/$USER/workspace/bazel/db --server-address=localhost:12100 directory save --root /Users/$USER/workspace/bazel/dir "**"
89a9068bd2d2784d5379b9fa3d02f02d9d0d7ecf4998a8e45b3d6784aacad4d4 157

4:

bzutil upload_action --cas_addr=localhost:12100 --command="1833d7c57656d2b7ee97e2068ce742f80e61357fba12a8b8d627782da3a58c29/11" --input_root="89a9068bd2d2784d5379b9fa3d02f02d9d0d7ecf4998a8e45b3d6784aacad4d4/157"
INFO[0000] Wrote to CAS successfully                     sourceLine="bzutil/main.go:220"
0cb08d1ec25eeed4d7a10d8a2cce85ea7810b76cfd43926e305288b19cf9aa3c/141

5:

bzutil execute --action="0cb08d1ec25eeed4d7a10d8a2cce85ea7810b76cfd43926e305288b19cf9aa3c/141"
INFO[0000] Operation: b53b5e99-0162-43e1-6cbd-e44f8e6d1566
	Done: false
	Metadata:
		Stage: QUEUED
		ActionDigest: 905f62f396fd837d111ac96007e38502736799a98f7207f90460004667e7fe25/138
  sourceLine="bzutil/main.go:236"

6:

bzutil get_operation --name="b53b5e99-0162-43e1-6cbd-e44f8e6d1566"
INFO[0000] Operation: b53b5e99-0162-43e1-6cbd-e44f8e6d1566
	Done: true
	Metadata:
		Stage: COMPLETED
		ActionDigest: 0cb08d1ec25eeed4d7a10d8a2cce85ea7810b76cfd43926e305288b19cf9aa3c/141
	ExecResponse:
		Status:
			Code: OK
		Cached: false
		ActionResult:
			ExitCode: 0
			OutputFiles: []
			OutputDirectories: []
			StdoutDigest: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0
			StderrDigest: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0
			ExecutionMetadata:
				Worker: bazel-worker
				QueueLatency: 420ms
				WorkerTotal: 17078ms
				InputFetch: 35ms
				Execution: 17031ms
				OutputUpload: 3ms
  sourceLine="bzutil/main.go:246"

GRPC through a proxy

If your GRPC-serving binaries are only accessible via a proxy, it is possible to redirect GRPC client commands via:

ssh <proxy-host> -L <local-proxy-port>:<service-host>:<service-port> -N &
grpc_cli call localhost:<local-proxy-port> ...

Documentation

Overview

Bazel Remote Execution API gRPC

Index

• Constants
• func DigestFromSnapshotID(id string) (*remoteexecution.Digest, error)
• func DigestFromString(s string) (*remoteexecution.Digest, error)
• func DigestStoreName(digest *remoteexecution.Digest) string
• func DigestToStr(d *remoteexecution.Digest) string
• func DigestsEqual(a, b *remoteexecution.Digest) bool
• func EmptyDigest() *remoteexecution.Digest
• func GetShaAndSize(id string) (string, int64, error)
• func IsEmptyDigest(d *remoteexecution.Digest) bool
• func IsValidDigest(hash string, size int64) bool
• func SnapshotID(sha string, size int64) string
• func SnapshotIDFromDigest(d *remoteexecution.Digest) string
• func ValidateID(id string) error
• type GRPCConfig
  • func (c *GRPCConfig) NewGRPCServer() *grpc.Server
  • func (c *GRPCConfig) NewListener() (net.Listener, error)
• type GRPCServer
• type TapLimiter
  • func NewTap(maxRequests, maxBurst int) *TapLimiter
  • func (t *TapLimiter) Handler(ctx context.Context, info *tap.Info) (context.Context, error)

Constants

const (
	// Instance-related constants
	DefaultInstanceName = ""

	// Resource storage-related constants
	StorePrefix = "blob"

	// Snapshot-related constants
	SnapshotIDPrefix = "bz"
	InvalidIDMsg     = "Expected ID to be of format bz-<sha256>-<sizeBytes>, was"

	// Nil-data/Empty SHA-256 data
	EmptySha  = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
	EmptySize = int64(0)

	// GRPC Server connection-related setting limits (defaults)
	MaxSimultaneousConnections = 0 // limits total simultaneous connections via the Listener
	MaxRequestsPerSecond       = 0 // limits total incoming requests allowed per second
	MaxRequestsBurst           = 0 // allows this many requests in a burst faster than MaxRPS average
	MaxConcurrentStreams       = 0 // limits concurrent streams _per client_
	MaxConnIdleMins            = 1 // limits open idle connections before the server closes them

	// Batch API max combined blobs inlined request size
	BatchMaxCombinedSize = 10 * 1024 * 1024
)

Functions

func DigestFromSnapshotID

func DigestFromSnapshotID(id string) (*remoteexecution.Digest, error)

Get a remoteexecution Digest from SnapshotID

func DigestFromString

func DigestFromString(s string) (*remoteexecution.Digest, error)

Create a Digest from a proprietary string format: "<hash>/<size>". Returns a Digest if parsed and validated, or an error.

func DigestStoreName

func DigestStoreName(digest *remoteexecution.Digest) string

Translate a Bazel Digest into a unique resource name for use in a bundleStore Only use hash in store name, size is dropped

func DigestToStr

func DigestToStr(d *remoteexecution.Digest) string

func DigestsEqual

func DigestsEqual(a, b *remoteexecution.Digest) bool

func EmptyDigest

func EmptyDigest() *remoteexecution.Digest

func GetShaAndSize

func GetShaAndSize(id string) (string, int64, error)

Get sha and size components from valid bazel SnapshotID

func IsEmptyDigest

func IsEmptyDigest(d *remoteexecution.Digest) bool

func IsValidDigest

func IsValidDigest(hash string, size int64) bool

Validate Digest hash and size components assuming SHA256 Size -1 indicates unknown size to support certain size-less operations

• this is a deviation from the public API

func SnapshotID

func SnapshotID(sha string, size int64) string

Generate a SnapshotID based on digest sha and size

func SnapshotIDFromDigest

func SnapshotIDFromDigest(d *remoteexecution.Digest) string

Generate a SnapshotID directly from a Digest

func ValidateID

func ValidateID(id string) error

Checks that ID is well formed

Types

type GRPCConfig

type GRPCConfig struct {
	GRPCAddr          string // Required: ip:port the Listener will bind to
	ListenerMaxConns  int    // Maximum simultaneous connections the listener will accept
	RateLimitPerSec   int    // Maximum incoming requests per second
	BurstLimitPerSec  int    // Maximum per-burst incoming requests per second (within RateLimitPerSec)
	ConcurrentStreams int    // Maximum concurrent GRPC streams allowed per client
	MaxConnIdleMins   int    // Maximum time a connection can remain open until the server closes it
	ConcurrentReqSize int64  // Size of resources used in concurrent requests, in bytes (Internal usage)
}

GRPCConfig holds fields used for configuring startup of GRPC Listeners and Servers Zero value integer fields are interpretted as unlimited

func (*GRPCConfig) NewGRPCServer

func (c *GRPCConfig) NewGRPCServer() *grpc.Server

Creates a new *grpc.Server configured with ServerOptions based on the GRPCConfig fields

func (*GRPCConfig) NewListener

func (c *GRPCConfig) NewListener() (net.Listener, error)

Creates a new net.Listener with the configured address and limits

type GRPCServer

type GRPCServer interface {
	Serve() error
}

gRPC server interface encapsulating gRPC operations and execution server, intended to reduce gRPC listener and registration boilerplate.

type TapLimiter

type TapLimiter struct {
	// contains filtered or unexported fields
}

Encapsulates a rate-per-second limiter that will check all incoming requests (per-connection goroutine) Handle func Fulfills grpc/tap.ServerInHandle

func NewTap

func NewTap(maxRequests, maxBurst int) *TapLimiter

Create a new TapLimiter with specified rate and burst allowance

func (*TapLimiter) Handler

func (t *TapLimiter) Handler(ctx context.Context, info *tap.Info) (context.Context, error)

Wait until the Limiter allows the a request or the Context expires Client sees non-nil err as an RPC error with code=Unavailable, desc includes RST_STREAM or REFUSED_STREAM