blobstore

Documentation

Overview

Package blobstore implements the REAPI ContentAddressableStorage and ByteStream services.

Index

• func FastCopy(source, destination string) error
• func LimitReadCloser(r io.ReadCloser, limit int64) io.ReadCloser
• func Register(s *grpc.Server, cas *ContentAddressableStorage, dataDir string) error
• type ContentAddressableStorage
  • func New(dataDir string) (*ContentAddressableStorage, error)
  • func (c *ContentAddressableStorage) Adopt(d digest.Digest, srcPath string) error
  • func (c *ContentAddressableStorage) Get(d digest.Digest) ([]byte, error)
  • func (c *ContentAddressableStorage) Has(d digest.Digest) bool
  • func (c *ContentAddressableStorage) LinkTo(d digest.Digest, path string) error
  • func (c *ContentAddressableStorage) Open(d digest.Digest, offset int64, limit int64) (io.ReadCloser, error)
  • func (c *ContentAddressableStorage) Proto(d digest.Digest, m proto.Message) error
  • func (c *ContentAddressableStorage) Put(data []byte) (digest.Digest, error)
  • func (c *ContentAddressableStorage) Stat(d digest.Digest) (os.FileInfo, error)
• type LimitedReadCloser
• type MissingBlobsError
  • func (e *MissingBlobsError) Error() string
• type Service
  • func NewService(cas *ContentAddressableStorage, uploadDir string) (*Service, error)
  • func (s *Service) BatchReadBlobs(ctx context.Context, request *repb.BatchReadBlobsRequest) (*repb.BatchReadBlobsResponse, error)
  • func (s *Service) BatchUpdateBlobs(ctx context.Context, request *repb.BatchUpdateBlobsRequest) (*repb.BatchUpdateBlobsResponse, error)
  • func (s *Service) FindMissingBlobs(ctx context.Context, request *repb.FindMissingBlobsRequest) (*repb.FindMissingBlobsResponse, error)
  • func (s *Service) GetTree(request *repb.GetTreeRequest, ...) error
  • func (s *Service) QueryWriteStatus(ctx context.Context, request *bspb.QueryWriteStatusRequest) (*bspb.QueryWriteStatusResponse, error)
  • func (s *Service) Read(request *bspb.ReadRequest, server bspb.ByteStream_ReadServer) error
  • func (s *Service) Write(server bspb.ByteStream_WriteServer) error

Functions

func FastCopy

func FastCopy(source, destination string) error

FastCopy copies a file from source to destination using a hard link. This is usually the best we can do, unless the operating system supports copy-on-write semantics for files (e.g. macOS with APFS).

func LimitReadCloser

func LimitReadCloser(r io.ReadCloser, limit int64) io.ReadCloser

LimitReadCloser wraps an io.LimitedReader in a LimitedReadCloser.

func Register

func Register(s *grpc.Server, cas *ContentAddressableStorage, dataDir string) error

Register creates and registers a new Service with the given gRPC server. The dataDir is created if it does not exist.

Types

type ContentAddressableStorage

type ContentAddressableStorage struct {
	// contains filtered or unexported fields
}

ContentAddressableStorage is a simple CAS implementation that stores files on the local disk.

func New

func New(dataDir string) (*ContentAddressableStorage, error)

New creates a new local CAS. The data directory is created if it does not exist.

func (*ContentAddressableStorage) Adopt

func (c *ContentAddressableStorage) Adopt(d digest.Digest, srcPath string) error

Adopt moves a file from the given path into the CAS. The digest is assumed to have been validated by the caller.

func (*ContentAddressableStorage) Get

func (c *ContentAddressableStorage) Get(d digest.Digest) ([]byte, error)

Get reads a file for the given digest from disk and returns its contents.

func (*ContentAddressableStorage) Has

func (c *ContentAddressableStorage) Has(d digest.Digest) bool

Has returns true if the requested digest exists in the CAS.

func (*ContentAddressableStorage) LinkTo

func (c *ContentAddressableStorage) LinkTo(d digest.Digest, path string) error

LinkTo creates a link `path` pointing to the file with digest `d` in the CAS. If the operating system supports cloning files via copy-on-write semantics, the file is cloned instead of hard linked.

func (*ContentAddressableStorage) Open

func (c *ContentAddressableStorage) Open(d digest.Digest, offset int64, limit int64) (io.ReadCloser, error)

Open returns an io.ReadCloser for the requested digest if it exists. The returned ReadCloser is limited to the given offset and limit. The offset must be non-negative and no larger than the file size. A limit of 0 means no limit, and a limit that's larger than the file size is truncated to the file size.

func (*ContentAddressableStorage) Proto

func (c *ContentAddressableStorage) Proto(d digest.Digest, m proto.Message) error

Proto reads a proto message with the given digest from the CAS and unmarshals it into m.

func (*ContentAddressableStorage) Put

func (c *ContentAddressableStorage) Put(data []byte) (digest.Digest, error)

Put stores the given data in the CAS and returns its digest.

func (*ContentAddressableStorage) Stat

func (c *ContentAddressableStorage) Stat(d digest.Digest) (os.FileInfo, error)

Stat returns os.FileInfo for the requested digest if it exists.

type LimitedReadCloser

type LimitedReadCloser struct {
	*io.LimitedReader
	io.Closer
}

LimitedReadCloser is an io.ReadCloser that limits the number of bytes that can be read.

type MissingBlobsError

type MissingBlobsError struct {
	// Blobs is the list of missing blobs.
	Blobs []digest.Digest
}

MissingBlobsError is an error type that indicates that one or more blobs are missing from the blob store. This is used to indicate that a client needs to upload the missing blobs before the operation can proceed.

func (*MissingBlobsError) Error

func (e *MissingBlobsError) Error() string

Error implements the error interface for MissingBlobsError so that it can be used as an error value.

type Service

type Service struct {
	repb.UnimplementedContentAddressableStorageServer
	bspb.UnimplementedByteStreamServer
	// contains filtered or unexported fields
}

Service implements the REAPI ContentAddressableStorage and ByteStream services.

func NewService

func NewService(cas *ContentAddressableStorage, uploadDir string) (*Service, error)

NewService creates a new Service.

func (*Service) BatchReadBlobs

func (s *Service) BatchReadBlobs(ctx context.Context, request *repb.BatchReadBlobsRequest) (*repb.BatchReadBlobsResponse, error)

func (*Service) BatchUpdateBlobs

func (s *Service) BatchUpdateBlobs(ctx context.Context, request *repb.BatchUpdateBlobsRequest) (*repb.BatchUpdateBlobsResponse, error)

BatchUpdateBlobs implements the ContentAddressableStorage.BatchUpdateBlobs RPC.

func (*Service) FindMissingBlobs

func (s *Service) FindMissingBlobs(ctx context.Context, request *repb.FindMissingBlobsRequest) (*repb.FindMissingBlobsResponse, error)

FindMissingBlobs implements the ContentAddressableStorage.FindMissingBlobs RPC.

func (*Service) GetTree

func (s *Service) GetTree(request *repb.GetTreeRequest, treeServer repb.ContentAddressableStorage_GetTreeServer) error

func (*Service) QueryWriteStatus

func (s *Service) QueryWriteStatus(ctx context.Context, request *bspb.QueryWriteStatusRequest) (*bspb.QueryWriteStatusResponse, error)

QueryWriteStatus implements the ByteStream.QueryWriteStatus RPC.

func (*Service) Read

func (s *Service) Read(request *bspb.ReadRequest, server bspb.ByteStream_ReadServer) error

Read implements the ByteStream.Read RPC.

func (*Service) Write

func (s *Service) Write(server bspb.ByteStream_WriteServer) error

Write implements the ByteStream.Write RPC.