cas

Documentation

Overview

Bazel Remote Execution API gRPC server Contains limited implementation of the ContentAddressableStore API interface

Index

• Constants
• Variables
• func GetDefaultReadResourceName(hash string, size int64) (string, error)
• func GetDefaultWriteResourceName(uuid, hash string, size int64) (string, error)
• func GetReadResourceName(instance, hash string, size int64, fname string) (string, error)
• func GetWriteResourceName(instance, uuid, hash string, size int64, fname string) (string, error)
• func IsNotFoundError(err error) bool
• func MakeCASServer(gc *bazel.GRPCConfig, sc *store.StoreConfig, stat stats.StatsReceiver) *casServer
• func MakeCASpbClient(cc conn.ClientConnPtr) remoteexecution.ContentAddressableStorageClient
• type BatchUploadContent
• type CASClient
  • func MakeCASClient() *CASClient
  • func (casCli *CASClient) BatchRead(r dialer.Resolver, digests []*remoteexecution.Digest, b backoff.BackOff) (map[string][]byte, error)
  • func (casCli *CASClient) BatchUpdateWrite(r dialer.Resolver, contents []BatchUploadContent, b backoff.BackOff) ([]*remoteexecution.Digest, error)
  • func (casCli *CASClient) ByteStreamRead(r dialer.Resolver, digest *remoteexecution.Digest, b backoff.BackOff) (bytes []byte, err error)
  • func (casCli *CASClient) ByteStreamWrite(r dialer.Resolver, digest *remoteexecution.Digest, data []byte, ...) (err error)
  • func (casCli *CASClient) FindMissingBlobs(r dialer.Resolver, digests []*remoteexecution.Digest, b backoff.BackOff) (missing []*remoteexecution.Digest, err error)
  • func (casCli *CASClient) GetCacheResult(r dialer.Resolver, digest *remoteexecution.Digest, b backoff.BackOff) (ar *remoteexecution.ActionResult, err error)
  • func (casCli *CASClient) SetCASpbMaker(...) *CASClient
  • func (casCli *CASClient) SetGrpcDialer(dialer conn.GRPCDialer) *CASClient
  • func (casCli *CASClient) UpdateCacheResult(r dialer.Resolver, digest *remoteexecution.Digest, ...) (out *remoteexecution.ActionResult, err error)
• type NotFoundError
  • func (e *NotFoundError) Error() string
• type Resource
  • func ParseReadResource(name string) (*Resource, error)
  • func ParseResource(instance, id, hash, sizeStr, name, format string) (*Resource, error)
  • func ParseWriteResource(name string) (*Resource, error)
  • func (r *Resource) String() string

Constants

const (
	// Resource naming constants
	ResourceNameType   = "blobs"
	ResourceNameAction = "uploads"

	// Default buffer sizes
	DefaultReadCapacity = 1024 * 1024

	// Batch parallelism for underlying store operations
	// NOTE experimental/arbitrary setting. Consider adding a mechanism to set via StoreConfigs
	// NOTE if service implementation is changed, retest with setting <= 5
	BatchParallelism = 5

	// ActionCache constants
	ResultAddressKey = "ActionCacheResult"

	// GRPC Server connection-related setting limits recommended for CAS
	MaxSimultaneousConnections = 1000 // limits total simultaneous connections via the Listener
	MaxRequestsPerSecond       = 500  // limits total incoming requests allowed per second
	MaxRequestsBurst           = 250  // allows this many requests in a burst faster than MaxRPS average
	MaxConcurrentStreams       = 0    // limits concurrent streams _per client_

)

Variables

var (
	// Resource naming format guidelines
	ResourceReadFormatStr  string = fmt.Sprintf("[<instance-name>/]%s/<hash>/<size>[/filename]", ResourceNameType)
	ResourceWriteFormatStr string = fmt.Sprintf("[<instance-name>/]%s/<uuid>/%s/<hash>/<size>[/filename]", ResourceNameAction, ResourceNameType)

	// Default TTL for CAS-based operations
	DefaultTTL time.Duration = time.Hour * 24 * 7

	// Duration to wait for available concurrent resources before returning an error
	WaitForResourceDuration time.Duration = time.Second * 10
)

Functions

func GetDefaultReadResourceName

func GetDefaultReadResourceName(hash string, size int64) (string, error)

func GetDefaultWriteResourceName

func GetDefaultWriteResourceName(uuid, hash string, size int64) (string, error)

func GetReadResourceName

func GetReadResourceName(instance, hash string, size int64, fname string) (string, error)

Return a valid read resource string based on individual components. Errors on invalid inputs.

func GetWriteResourceName

func GetWriteResourceName(instance, uuid, hash string, size int64, fname string) (string, error)

Return a valid write resource string based on individual components. Errors on invalid inputs

func IsNotFoundError

func IsNotFoundError(err error) bool

Returns true if an error is of type NotFoundError

func MakeCASServer

func MakeCASServer(gc *bazel.GRPCConfig, sc *store.StoreConfig, stat stats.StatsReceiver) *casServer

Creates a new GRPCServer (CASServer/ByteStreamServer/ActionCacheServer) based on GRPCConfig, StoreConfig, and StatsReceiver, and preregisters the service

func MakeCASpbClient

func MakeCASpbClient(cc conn.ClientConnPtr) remoteexecution.ContentAddressableStorageClient

Types

type BatchUploadContent

type BatchUploadContent struct {
	Digest *remoteexecution.Digest
	Data   []byte
}

type CASClient

type CASClient struct {
	CASpbMaker func(cc conn.ClientConnPtr) remoteexecution.ContentAddressableStorageClient // func that returns the proto client
	// contains filtered or unexported fields
}

CASClient struct with exposed 3rd party dependencies (the grpcDialer and function to create protobuf CAS client) to allow injecting mocks during testing

func MakeCASClient

func MakeCASClient() *CASClient

make a CASClient that uses the production 3rd party libraries. Testing will overwrite these values (using the setters below) with mocks

func (*CASClient) BatchRead

func (casCli *CASClient) BatchRead(r dialer.Resolver,
	digests []*remoteexecution.Digest, b backoff.BackOff) (map[string][]byte, error)

client for downloading a list of digests from the CAS BatchReadBlobs (as a single batch)

func (*CASClient) BatchUpdateWrite

func (casCli *CASClient) BatchUpdateWrite(r dialer.Resolver,
	contents []BatchUploadContent, b backoff.BackOff) ([]*remoteexecution.Digest, error)

client for uploading a list of digests to CAS BatchUpdateBlobs (as a single batch)

func (*CASClient) ByteStreamRead

func (casCli *CASClient) ByteStreamRead(r dialer.Resolver, digest *remoteexecution.Digest, b backoff.BackOff) (bytes []byte, err error)

Read data as bytes from a CAS. Takes a Resolver for addressing and a bazel Digest to read. Returns bytes read or an error. If the requested resource was not found, returns a NotFoundError

func (*CASClient) ByteStreamWrite

func (casCli *CASClient) ByteStreamWrite(r dialer.Resolver, digest *remoteexecution.Digest, data []byte, b backoff.BackOff) (err error)

Write data as bytes to a CAS. Takes a Resolver for addressing, a bazel Digest to read, and []byte data.

func (*CASClient) FindMissingBlobs

func (casCli *CASClient) FindMissingBlobs(r dialer.Resolver,
	digests []*remoteexecution.Digest, b backoff.BackOff) (missing []*remoteexecution.Digest, err error)

Make a FindMissingBlobs request to a CAS given a set of digests. Returns digests that are missing on the server.

func (*CASClient) GetCacheResult

func (casCli *CASClient) GetCacheResult(r dialer.Resolver,
	digest *remoteexecution.Digest, b backoff.BackOff) (ar *remoteexecution.ActionResult, err error)

Client function for GetActionResult requests. Takes a Resolver for ActionCache server and Digest to get.

func (*CASClient) SetCASpbMaker

func (casCli *CASClient) SetCASpbMaker(pcClientMaker func(cc conn.ClientConnPtr) remoteexecution.ContentAddressableStorageClient) *CASClient

func (*CASClient) SetGrpcDialer

func (casCli *CASClient) SetGrpcDialer(dialer conn.GRPCDialer) *CASClient

setters use builder pattern (y.Set(x) returns y) for more succinct construction

func (*CASClient) UpdateCacheResult

func (casCli *CASClient) UpdateCacheResult(r dialer.Resolver, digest *remoteexecution.Digest,
	ar *remoteexecution.ActionResult, b backoff.BackOff) (out *remoteexecution.ActionResult, err error)

Client function for UpdateActionResult requests. Takes a Resolver for ActionCache server and Digest/ActionResult to update.

type NotFoundError

type NotFoundError struct {
	Err string
}

Type that indicates a CAS client operation found because the server returned a GRPC NOT_FOUND error.

func (*NotFoundError) Error

func (e *NotFoundError) Error() string

Implements Error interface

type Resource

type Resource struct {
	Instance string
	Digest   *remoteexecution.Digest
	UUID     uuid.UUID
}

Keep track of a Resource specified by a client. Instance - optional parameter identifying a server instance Digest - Bazel Digest identifier UUID - client identifier attached to write requests

Unused by Scoot currently except for tracking/logging

func ParseReadResource

func ParseReadResource(name string) (*Resource, error)

Parses a name string from the Read API into a Resource for bazel artifacts. Valid read format: "[<instance>/]blobs/<hash>/<size>[/<filename>]" Scoot does not currently use/track the filename portion of resource names

func ParseResource

func ParseResource(instance, id, hash, sizeStr, name, format string) (*Resource, error)

Underlying Resource parser from separated URI components

func ParseWriteResource

func ParseWriteResource(name string) (*Resource, error)

Parses a name string from the Write API into a Resource for bazel artifacts. Valid read format: "[<instance>/]uploads/<uuid>/blobs/<hash>/<size>[/<filename>]" Scoot does not currently use/track the filename portion of resource names

func (*Resource) String

func (r *Resource) String() string