buffer

Documentation

Index

• Variables
• func WithBackgroundTask(b Buffer) (Buffer, *BackgroundTask)
• type BackgroundTask
  • func (t *BackgroundTask) Finish(err error)
• type Buffer
  • func NewBufferFromError(err error) Buffer
  • func NewCASBufferFromByteSlice(digest digest.Digest, data []byte, repairStrategy RepairStrategy) Buffer
  • func NewCASBufferFromChunkReader(digest digest.Digest, r ChunkReader, repairStrategy RepairStrategy) Buffer
  • func NewCASBufferFromReader(digest digest.Digest, r io.ReadCloser, repairStrategy RepairStrategy) Buffer
  • func NewProtoBufferFromByteSlice(m proto.Message, data []byte, repairStrategy RepairStrategy) Buffer
  • func NewProtoBufferFromProto(message proto.Message, repairStrategy RepairStrategy) Buffer
  • func NewProtoBufferFromReader(m proto.Message, r io.ReadCloser, repairStrategy RepairStrategy) Buffer
  • func NewValidatedBufferFromByteSlice(data []byte) Buffer
  • func NewValidatedBufferFromFileReader(r filesystem.FileReader, sizeBytes int64) Buffer
  • func WithErrorHandler(b Buffer, errorHandler ErrorHandler) Buffer
• type ChunkReader
• type ErrorHandler
• type RepairFunc
• type RepairStrategy
  • func Reparable(digest digest.Digest, repairFunc RepairFunc) RepairStrategy

Variables

var (
	// UserProvided indicates that the buffer did not come from
	// storage. Instead, it is an artifact that is currently being
	// uploaded by a user or automated process. When data
	// consistency errors occur, no data needs to be repaired. It is
	// sufficient to return an error to the user.
	UserProvided = RepairStrategy{
					// contains filtered or unexported fields
	}
	// Irreparable indicates that the buffer was obtained from
	// storage, but that the storage provides no method for
	// repairing the data. This doesn't necessarily have to be
	// harmful. It may well be the case that the storage backend
	// also has logic in place to detect inconsistencies and that
	// there is no need for us to report those.
	Irreparable = RepairStrategy{
				// contains filtered or unexported fields
	}
)

Functions

func WithBackgroundTask

func WithBackgroundTask(b Buffer) (Buffer, *BackgroundTask)

WithBackgroundTask returns a decorated Buffer which will at the end of its lifetime wait for the completion of a background task. The error of the background task is returned if the Buffer itself yields no other error.

This function may be used by implementations of BlobAccess that multiplex data, ensuring that the foreground goroutine will at some point block, waiting for background synchronizations to complete. This ensures that the number of concurrent goroutines remains bounded and that any synchronization errors can be propagated.

Types

type BackgroundTask

type BackgroundTask struct {
	// contains filtered or unexported fields
}

BackgroundTask is a handle returned by WithBackgroundTask(). The Finish() function must be called exactly once to indicate the background task associated with the buffer is completed.

func (*BackgroundTask) Finish

func (t *BackgroundTask) Finish(err error)

Finish the background task.

type Buffer

type Buffer interface {
	// Return the size of the data stored in the buffer. This
	// function may fail if the buffer is in a known error state in
	// which the size of the object is unknown.
	GetSizeBytes() (int64, error)

	// Write the entire contents of the buffer into a Writer.
	IntoWriter(w io.Writer) error
	// Read a part of the buffer into a byte slice.
	ReadAt(p []byte, off int64) (int, error)
	// Return the contents in the form of an unmarshaled
	// Protobuf message.
	//
	// If and only if the buffer isn't already backed by a Protobuf
	// message, the provided message is used to store the
	// unmarshaled message. The caller must use a type assertion to
	// convert this function's return value back to the appropriate
	// message type.
	ToProto(m proto.Message, maximumSizeBytes int) (proto.Message, error)
	// Return the full contents of the buffer as a byte slice.
	ToByteSlice(maximumSizeBytes int) ([]byte, error)
	// Read the contents of the buffer, starting at a given offset,
	// as a stream of byte slices. Normally used by the Content
	// Addressable Storage.
	ToChunkReader(off int64, maximumChunkSizeBytes int) ChunkReader
	// Obtain a reader that returns the entire contents of the
	// buffer.
	ToReader() io.ReadCloser
	// Obtain two handles to the same underlying object in such a
	// way that they may get copied. This function may be used when
	// buffers need to be inspected prior to returning them.
	CloneCopy(maximumSizeBytes int) (Buffer, Buffer)
	// Obtain two handles to the same underlying object in such a
	// way that the underlying stream may be multiplexed. This
	// function may be used if a single buffer is passed to multiple
	// goroutines that process the same data.
	//
	// It is not safe to use both buffers within same goroutine, as
	// this may cause deadlocks. Contents are only returned when
	// both buffers are accessed.
	CloneStream() (Buffer, Buffer)
	// Release the object without reading its contents.
	Discard()
	// contains filtered or unexported methods
}

Buffer of data to be read from/written to the Action Cache (AC) or Content Addressable Storage (CAS).

BlobAccess implementations gain access to data through a variety of APIs. Some of these APIs present data as slices of bytes, while others provide io.{Reader,ReadCloser} objects. Similarly, on the consumer side, data may also need to be read/written in different formats.

Buffers are an abstraction on top of slices and readers, allowing data to be converted from one format to the other. It also attempts to prevent conversions/copying when unnecessary. For example, when creating a buffer from a slice of bytes, a call to ToByteSlice() will return the original slice.

Buffers also attempt to ensure the data is consistent. In the case of buffers created using NewProtoBufferFrom*(), data may only be extracted in case the provided data corresponds to a valid Protobuf message. In the case of buffers created using NewCASBufferFrom*(), data may only be extracted in case the size and checksum match the digest.

func NewBufferFromError

func NewBufferFromError(err error) Buffer

NewBufferFromError creates a Buffer that returns a fixed error response for all operations.

func NewCASBufferFromByteSlice

func NewCASBufferFromByteSlice(digest digest.Digest, data []byte, repairStrategy RepairStrategy) Buffer

NewCASBufferFromByteSlice creates a buffer for an object stored in the Content Addressable Storage, backed by a byte slice.

func NewCASBufferFromChunkReader

func NewCASBufferFromChunkReader(digest digest.Digest, r ChunkReader, repairStrategy RepairStrategy) Buffer

NewCASBufferFromChunkReader creates a buffer for an object stored in the Content Addressable Storage, backed by a ChunkReader.

func NewCASBufferFromReader

func NewCASBufferFromReader(digest digest.Digest, r io.ReadCloser, repairStrategy RepairStrategy) Buffer

NewCASBufferFromReader creates a buffer for an object stored in the Content Addressable Storage, whose contents may be obtained through a ReadCloser.

func NewProtoBufferFromByteSlice

func NewProtoBufferFromByteSlice(m proto.Message, data []byte, repairStrategy RepairStrategy) Buffer

NewProtoBufferFromByteSlice creates a buffer for an object contained in a Protobuf storage such as Action Cache, based on a marshaled Protobuf message stored in a byte slice. An empty Protobuf message object must be provided that corresponds with the type of the data contained in the byte slice.

func NewProtoBufferFromProto

func NewProtoBufferFromProto(message proto.Message, repairStrategy RepairStrategy) Buffer

NewProtoBufferFromProto creates a buffer for an object contained in a Protobuf storage such as the Action Cache, based on an unmarshaled Protobuf message.

func NewProtoBufferFromReader

func NewProtoBufferFromReader(m proto.Message, r io.ReadCloser, repairStrategy RepairStrategy) Buffer

NewProtoBufferFromReader creates a buffer for an object contained in a Protobuf storage such as the Action Cache, based on a marshaled Protobuf message that may be obtained through a ReadCloser. An empty Protobuf message object must be provided that corresponds with the type of the data contained in the reader.

func NewValidatedBufferFromByteSlice

func NewValidatedBufferFromByteSlice(data []byte) Buffer

NewValidatedBufferFromByteSlice creates a Buffer that is backed by a slice of bytes. No checking of data integrity is performed, as it is assumed that the data stored in the slice is valid.

func NewValidatedBufferFromFileReader

func NewValidatedBufferFromFileReader(r filesystem.FileReader, sizeBytes int64) Buffer

NewValidatedBufferFromFileReader creates a Buffer that is backed by a FileReader. No checking of data integrity is performed, as it is assumed that the data stored in the slice is valid.

This function should be used with care, as media backing FileReaders (e.g., local file systems, block devices) may well be prone to data corruption. This will not be detected if buffers are constructed using this function.

The provided FileReader must permit ReadAt() to be called in parallel, as cloning the buffer may permit multiple goroutines to access the data.

func WithErrorHandler

func WithErrorHandler(b Buffer, errorHandler ErrorHandler) Buffer

WithErrorHandler attaches an ErrorHandler to a Buffer. If the provided Buffer is already in a guaranteed success/failure state, the ErrorHandler may be applied immediately.

type ChunkReader

type ChunkReader interface {
	Read() ([]byte, error)
	Close()
}

ChunkReader is similar to io.ReadCloser, except that data is not copied from the stream into an output array. The implementation is responsible for providing space for the data. This interface is similar to how frame-based transfer protocols work, including the Bytestream protocol used by REv2.

type ErrorHandler

type ErrorHandler interface {
	OnError(err error) (Buffer, error)
	Done()
}

ErrorHandler provides the possibility to hook into errors that occur on Buffer objects. There are three different use cases:

• To capture errors that occur on buffers, as done by MetricsBlobAccess.
• To substitute or augment errors that occur on buffers, as done by ExistencePreconditionBlobAccess.
• To provide an alternative buffer where the desired content may be obtained instead. This may be useful for storage backends capable of replication.

Every instance of ErrorHandler will have zero or more calls against OnError() performed against it, followed by a call to Done().

ErrorHandler is only used by Buffer implementations in error scenarios where the transmission may continue if another buffer was provided, such as I/O errors. There are some errors conditions where ErrorHandler is not used, such as:

• Passing a negative offset to ToChunkReader(). Switching to another buffer would not yield any further progress.
• Checksum mismatches on streams returned by ToChunkReader() and ToReader(). The faulty data may have already been returned to the consumer of the buffer through Read(), which cannot be undone by switching to another buffer. Checksum mismatches returned by ReadAt() and ToByteSlice() will be caught, as those can be retried in their entirety.

type RepairFunc

type RepairFunc func() error

RepairFunc is a callback that may be invoked by buffer objects to report that the contents of the buffer are observed to be invalid. More concretely, it is invoked when an Protobuf buffer object could not be unmarshaled into a Protobuf message, or when the contents of a Content Addressable Storage buffer don't match up with the digest of the object.

This callback allows storage backends to repair the object, for example by replicating it from other storage backends or by simply deleting it, so that it may be recomputed or re-uploaded.

type RepairStrategy

type RepairStrategy struct {
	// contains filtered or unexported fields
}

RepairStrategy is passed to most New*Buffer() creation functions to specify a strategy for how to deal with data consistency issues.

func Reparable

func Reparable(digest digest.Digest, repairFunc RepairFunc) RepairStrategy

Reparable indicates that the buffer was obtained from storage and that the storage backend provides a method for repairing data inconsistencies. A callback is provided to trigger repairs.