blobstore

Documentation

Index

• func CASPutProto(ctx context.Context, blobAccess BlobAccess, message proto.Message, ...) (digest.Digest, error)
• type BlobAccess
  • func NewCloudBlobAccess(bucket *blob.Bucket, keyPrefix string, storageType StorageType) BlobAccess
  • func NewEmptyBlobInjectingBlobAccess(base BlobAccess) BlobAccess
  • func NewErrorBlobAccess(err error) BlobAccess
  • func NewExistenceCachingBlobAccess(base BlobAccess, existenceCache *digest.ExistenceCache) BlobAccess
  • func NewMetricsBlobAccess(blobAccess BlobAccess, clock clock.Clock, name string) BlobAccess
  • func NewReadFallbackBlobAccess(primary BlobAccess, secondary BlobAccess) BlobAccess
  • func NewRedisBlobAccess(redisClient RedisClient, storageType StorageType, keyTTL time.Duration, ...) BlobAccess
  • func NewReferenceExpandingBlobAccess(blobAccess BlobAccess, httpClient HTTPClient, s3 cloud_aws.S3, ...) BlobAccess
  • func NewRemoteBlobAccess(address string, prefix string, storageType StorageType) BlobAccess
  • func NewSizeDistinguishingBlobAccess(smallBlobAccess BlobAccess, largeBlobAccess BlobAccess, cutoffSizeBytes int64) BlobAccess
• type HTTPClient
• type RedisClient
• type StorageType

Functions

func CASPutProto

func CASPutProto(ctx context.Context, blobAccess BlobAccess, message proto.Message, parentDigest digest.Digest) (digest.Digest, error)

CASPutProto is a helper function for storing Protobuf messages in the Content Addressable Storage (CAS). It computes the digest of the message and stores it under that key. The digest is then returned, so that the object may be referenced.

Types

type BlobAccess

type BlobAccess interface {
	Get(ctx context.Context, digest digest.Digest) buffer.Buffer
	Put(ctx context.Context, digest digest.Digest, b buffer.Buffer) error
	FindMissing(ctx context.Context, digests digest.Set) (digest.Set, error)
}

BlobAccess is an abstraction for a data store that can be used to hold both a Bazel Action Cache (AC) and Content Addressable Storage (CAS).

func NewCloudBlobAccess

func NewCloudBlobAccess(bucket *blob.Bucket, keyPrefix string, storageType StorageType) BlobAccess

NewCloudBlobAccess creates a BlobAccess that uses a cloud-based blob storage as a backend.

func NewEmptyBlobInjectingBlobAccess

func NewEmptyBlobInjectingBlobAccess(base BlobAccess) BlobAccess

NewEmptyBlobInjectingBlobAccess is a decorator for BlobAccess that causes it to directly process any requests for blobs of size zero. Get() operations immediately return an empty buffer, while Put() operations for such buffers are ignored.

Bazel never attempts to read the empty blob from the Content Addressable Storage, which by itself is harmless. In addition to that, it never attempts to write the empty blob. This is problematic, as it may cause unaware implementations of GetActionResult() and input root population to fail.

This problem remained undetected for a long time, because running at least one build action through bb_worker has a high probability of creating the empty blob in storage explicitly.

The consensus within the Remote APIs working group has been to give the empty blob a special meaning: the system must behave as if this blob is always present.

More details: https://github.com/bazelbuild/bazel/issues/11063

func NewErrorBlobAccess

func NewErrorBlobAccess(err error) BlobAccess

NewErrorBlobAccess creates a BlobAccess that returns a fixed error response. Such an implementation is useful for adding explicit rejection of oversized requests or disabling storage entirely.

func NewExistenceCachingBlobAccess

func NewExistenceCachingBlobAccess(base BlobAccess, existenceCache *digest.ExistenceCache) BlobAccess

NewExistenceCachingBlobAccess creates a decorator for BlobAccess that adds caching to the FindMissing() operation.

Clients such as Bazel tend to frequently call ContentAddressableStorage.FindMissingBlobs() with overlapping sets of digests. They don't seem to have a local cache of which digests they queried recently. This decorator adds such a cache.

This decorator may be useful to run on instances that act as frontends for a mirrored/sharding storage pool, as it may reduce the load observed on the storage pool.

func NewMetricsBlobAccess

func NewMetricsBlobAccess(blobAccess BlobAccess, clock clock.Clock, name string) BlobAccess

NewMetricsBlobAccess creates an adapter for BlobAccess that adds basic instrumentation in the form of Prometheus metrics.

func NewReadFallbackBlobAccess

func NewReadFallbackBlobAccess(primary BlobAccess, secondary BlobAccess) BlobAccess

NewReadFallbackBlobAccess creates a decorator for BlobAccess that causes reads for non-existent to be forwarded to a secondary storage backend. Data is never written to the latter.

This decorator can be used to integrate external data sets into the system, e.g. by combining it with ReferenceExpandingBlobAccess.

func NewRedisBlobAccess

func NewRedisBlobAccess(redisClient RedisClient,
	storageType StorageType,
	keyTTL time.Duration,
	replicationCount int64,
	replicationTimeout time.Duration) BlobAccess

NewRedisBlobAccess creates a BlobAccess that uses Redis as its backing store.

func NewReferenceExpandingBlobAccess

func NewReferenceExpandingBlobAccess(blobAccess BlobAccess, httpClient HTTPClient, s3 cloud_aws.S3, maximumMessageSizeBytes int) BlobAccess

NewReferenceExpandingBlobAccess takes an Indirect Content Addressable Storage (ICAS) backend and converts it to a Content Addressable Storage (CAS) backend. Any object requested through this BlobAccess will cause its reference to be loaded from the ICAS, followed by fetching its data from the referenced location.

func NewRemoteBlobAccess

func NewRemoteBlobAccess(address string, prefix string, storageType StorageType) BlobAccess

NewRemoteBlobAccess for use of HTTP/1.1 cache backend.

See: https://docs.bazel.build/versions/master/remote-caching.html#http-caching-protocol

func NewSizeDistinguishingBlobAccess

func NewSizeDistinguishingBlobAccess(smallBlobAccess BlobAccess, largeBlobAccess BlobAccess, cutoffSizeBytes int64) BlobAccess

NewSizeDistinguishingBlobAccess creates a BlobAccess that splits up requests between two backends based on the size of the object specified in the digest. Backends tend to have different performance characteristics based on blob size. This adapter may be used to optimize performance based on that.

type HTTPClient

type HTTPClient interface {
	Do(req *http.Request) (*http.Response, error)
}

HTTPClient is an interface around Go's standard HTTP client type. It has been added to aid unit testing.

type RedisClient

type RedisClient interface {
	redis.Cmdable
	Process(cmd redis.Cmder) error
}

RedisClient is an interface that contains the set of functions of the Redis library that is used by this package. This permits unit testing and uniform switching between clustered and single-node Redis.

type StorageType

type StorageType interface {
	// GetDigestKey creates a string key that may be used as an
	// identifier for a blob. This function is, for example, used to
	// determine the name of keys in S3 and Redis.
	GetDigestKey(digest digest.Digest) string

	// NewBufferFromByteSlice creates a buffer from a byte slice
	// that is either suitable for storage in the CAS or AC.
	NewBufferFromByteSlice(digest digest.Digest, data []byte, repairStrategy buffer.RepairStrategy) buffer.Buffer
	// NewBufferFromReader creates a buffer from a reader that is
	// either suitable for storage in the CAS or AC.
	NewBufferFromReader(digest digest.Digest, r io.ReadCloser, repairStrategy buffer.RepairStrategy) buffer.Buffer
}

StorageType is passed to many implementations of BlobAccess to be able to use the same BlobAccess implementation for both the Content Addressable Storage (CAS) and Action Cache (AC). This interface provides functions for generic object keying and buffer creation.

var ACStorageType StorageType = acStorageType{}

ACStorageType is capable of creating identifiers and buffers for objects stored in the Action Cache (AC).

var CASStorageType StorageType = casStorageType{}

CASStorageType is capable of creating identifiers and buffers for objects stored in the Content Addressable Storage (CAS).

var ICASStorageType StorageType = icasStorageType{}

ICASStorageType is capable of creating identifiers and buffers for objects stored in the Indirect Content Addressable Storage (ICAS).