Documentation
¶
Overview ¶
Package cas implements an efficient client for Content Addressable Storage.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrSkip when returned by UploadOptions.Prelude, means the file/dir must be // not be uploaded. // // Note that if UploadOptions.PreserveSymlinks is true and the ErrSkip is // returned for a symlink target, but not the symlink itself, then it may // result in a dangling symlink. ErrSkip = errors.New("skip file") // ErrDigestUnknown indicates that the requested digest is unknown. // Use errors.Is instead of direct equality check. ErrDigestUnknown = errors.New("the requested digest is unknown") )
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// InstanceName is the full name of the RBE instance.
InstanceName string
// Config is the configuration that the client was created with.
Config ClientConfig
// contains filtered or unexported fields
}
Client is a client for Content Addressable Storage. Create one using NewClient.
Goroutine-safe.
All fields are considered immutable, and should not be changed.
func NewClient ¶
func NewClient(ctx context.Context, conn grpc.ClientConnInterface, instanceName string) (*Client, error)
NewClient creates a new client with the default configuration. Use client.Dial to create a connection.
func NewClientWithConfig ¶
func NewClientWithConfig(ctx context.Context, conn grpc.ClientConnInterface, instanceName string, config ClientConfig) (*Client, error)
NewClientWithConfig creates a new client and accepts a configuration.
func (*Client) Upload ¶
func (c *Client) Upload(ctx context.Context, opt UploadOptions, inputC <-chan *UploadInput) (*UploadResult, error)
Upload uploads all files/directories specified by inputC.
Upload assumes ownership of UploadInputs received from inputC. They must not be mutated after sending.
Close inputC to indicate that there are no more files/dirs to upload. When inputC is closed, Upload finishes uploading the remaining files/dirs and exits successfully.
If ctx is canceled, the Upload returns with an error.
type ClientConfig ¶
type ClientConfig struct {
// FSConcurrency is the maximum number of concurrent file system operations.
// TODO(nodir): ensure this does not hurt streaming performance
FSConcurrency int
// FSLargeConcurrency is the maximum number of concurrent large file read operation.
FSLargeConcurrency int
// SmallFileThreshold is a size threshold to categorize a file as small.
// Such files are buffered entirely (read only once).
SmallFileThreshold int64
// LargeFileThreshold is a size threshold to categorize a file as large. For
// such files, IO concurrency limits are much tighter and locality is
// prioritized: the file is read for the first and second times with minimal
// delay between the two.
LargeFileThreshold int64
// FileIOSize is the size of file reads.
FileIOSize int64
// CompressedBytestreamThreshold is the minimum blob size to enable compression
// in ByteStream RPCs.
// Use 0 for all writes being compressed, and a negative number for all operations being
// uncompressed.
// DefaultClientConfig() disables compression by default.
CompressedBytestreamThreshold int64
// FindMissingBlobs is configuration for ContentAddressableStorage.FindMissingBlobs RPCs.
// FindMissingBlobs.MaxSizeBytes is ignored.
FindMissingBlobs RPCConfig
// BatchUpdateBlobs is configuration for ContentAddressableStorage.BatchUpdateBlobs RPCs.
BatchUpdateBlobs RPCConfig
// ByteStreamWrite is configuration for ByteStream.Write RPCs.
// ByteStreamWrite.MaxItems is ignored.
ByteStreamWrite RPCConfig
// RetryPolicy specifies how to retry requests on transient errors.
RetryPolicy retry.BackoffPolicy
// IgnoreCapabilities specifies whether to ignore server-provided capabilities.
// Capabilities are consulted by default.
IgnoreCapabilities bool
}
ClientConfig is a config for Client. See DefaultClientConfig() for the default values.
func DefaultClientConfig ¶
func DefaultClientConfig() ClientConfig
DefaultClientConfig returns the default config.
To override a specific value:
cfg := DefaultClientConfig() ... mutate cfg ... client, err := NewClientWithConfig(ctx, cfg)
func (*ClientConfig) Validate ¶
func (c *ClientConfig) Validate() error
Validate returns a non-nil error if the config is invalid.
type DigestStat ¶
type DigestStat struct {
Digests int64 // number of unique digests
Bytes int64 // total sum of digest sizes
}
DigestStat is aggregated statistics over a set of digests.
type RPCConfig ¶
type RPCConfig struct {
// Concurrency is the maximum number of RPCs in flight.
Concurrency int
// MaxSizeBytes is the maximum size of the request/response, in bytes.
MaxSizeBytes int
// MaxItems is the maximum number of blobs/digests per RPC.
// Applies only to batch RPCs, such as FindMissingBlobs.
MaxItems int
// Timeout is the maximum duration of the RPC.
Timeout time.Duration
}
RPCConfig is configuration for a particular CAS RPC. Some of the fields might not apply to certain RPCs.
For streaming RPCs, the values apply to individual requests/responses in a stream, not the entire stream.
type TransferStats ¶
type TransferStats struct {
CacheHits DigestStat
CacheMisses DigestStat
Streamed DigestStat // streamed transfers
Batched DigestStat // batched transfers
}
TransferStats is upload/download statistics.
type UploadInput ¶
type UploadInput struct {
// Path to the file or a directory to upload.
// Must be absolute.
Path string
// Allowlist is a filter for files/directories under Path.
// If a file is not a present in Allowlist and does not reside in a directory
// present in the Allowlist, then the file is ignored.
// This is equivalent to deleting all not-matched files/dirs before
// uploading.
//
// Each path in the Allowlist must be relative to UploadInput.Path.
//
// Must be empty if Path points to a regular file.
Allowlist []string
// Exclude is a file/dir filter. If Exclude is not nil and the
// absolute path of a file/dir match this regexp, then the file/dir is skipped.
// Forward-slash-separated paths are matched against the regexp: PathExclude
// does not have to be conditional on the OS.
// If the Path is a directory, then the filter is evaluated against each file
// in the subtree.
// See ErrSkip comments for more details on semantics regarding excluding symlinks .
Exclude *regexp.Regexp
// contains filtered or unexported fields
}
UploadInput specifies a file or directory to upload.
func (*UploadInput) Digest ¶
func (in *UploadInput) Digest(relPath string) (digest.Digest, error)
Digest returns the digest computed for a file/dir. The relPath is relative to UploadInput.Path. Use "." for the digest of the UploadInput.Path itself.
Digest is safe to call only after the channel returned by DigestsComputed() is closed.
If the digest is unknown, returns (nil, err), where err is ErrDigestUnknown according to errors.Is. If the file is a danging symlink, then its digest is unknown.
func (*UploadInput) DigestsComputed ¶
func (in *UploadInput) DigestsComputed() <-chan struct{}
DigestsComputed returns a channel which is closed when all digests, including descendants, are computed. It is guaranteed to be closed by the time Client.Upload() returns successfully.
DigestsComputed() is always safe to call.
type UploadOptions ¶
type UploadOptions struct {
// PreserveSymlinks specifies whether to preserve symlinks or convert them
// to regular files. This doesn't upload target of symlinks, caller needs
// to specify targets explicitly if those are necessary too.
PreserveSymlinks bool
// AllowDanglingSymlinks specifies whether to upload dangling links or halt
// the upload with an error.
//
// This field is ignored if PreserveSymlinks is false, which is the default.
AllowDanglingSymlinks bool
// Prelude is called for each file/dir to be read and uploaded.
// If it returns an error which is ErrSkip according to errors.Is, then the
// file/dir is not processed.
// If it returns another error, then the upload is halted with that error.
//
// Prelude might be called multiple times for the same file if different
// UploadInputs directly/indirectly refer to the same file, but with different
// UploadInput.Exclude.
//
// Prelude is called from different goroutines.
Prelude func(absPath string, mode os.FileMode) error
}
UploadOptions is optional configuration for Upload function. The default options are the zero value of this struct.
type UploadResult ¶
type UploadResult struct {
Stats TransferStats
// contains filtered or unexported fields
}
UploadResult is the result of a Client.Upload call. It provides file/dir digests and statistics.
Source Files