circular

Documentation

Index

• func NewCircularBlobAccess(storageType string, offsetStore OffsetStore, dataStore DataStore, ...) blobstore.BlobAccess
• type Cursors
  • func (c *Cursors) Contains(offset uint64, length int64) bool
• type DataStore
  • func NewFileDataStore(file ReadWriterAt, size uint64) DataStore
• type OffsetStore
  • func NewCachingOffsetStore(backend OffsetStore, size uint) OffsetStore
  • func NewDemultiplexingOffsetStore(offsetStoreGetter OffsetStoreGetter) OffsetStore
  • func NewFileOffsetStore(storageType string, file ReadWriterAt, size uint64) OffsetStore
• type OffsetStoreGetter
• type ReadWriterAt
• type StateStore
  • func NewBulkAllocatingStateStore(stateStore StateStore, chunkSize uint64) StateStore
  • func NewFileStateStore(file ReadWriterAt, dataSize uint64) (StateStore, error)
  • func NewPositiveSizedBlobStateStore(stateStore StateStore) StateStore

Functions

func NewCircularBlobAccess

func NewCircularBlobAccess(storageType string, offsetStore OffsetStore, dataStore DataStore, stateStore StateStore) blobstore.BlobAccess

NewCircularBlobAccess creates a new circular storage backend. Instead of writing data to storage directly, all three storage files are injected through separate interfaces.

Types

type Cursors

type Cursors struct {
	Read  uint64
	Write uint64
}

Cursors is a pair of offsets within the data file, indicating which part of the file contains valid readable data and where future writes need to take place.

func (*Cursors) Contains

func (c *Cursors) Contains(offset uint64, length int64) bool

Contains returns whether the provided offset and length are contained with the cursors. In effect, it returns whether the offset/length pair still corresponds to valid data.

type DataStore

type DataStore interface {
	Put(r io.Reader, offset uint64) error
	Get(offset uint64, size int64) io.ReadCloser
}

DataStore is where the data corresponding with a blob is stored. Data can be accessed by providing an offset within the data store and its length.

func NewFileDataStore

func NewFileDataStore(file ReadWriterAt, size uint64) DataStore

NewFileDataStore creates a new file-based store for blob contents. All data is stored in a single file, where all blobs are concatenated directly. As the file pointer wraps around at a configured size, old data is automatically overwritten by new data.

type OffsetStore

type OffsetStore interface {
	Get(digest *util.Digest, cursors Cursors) (uint64, int64, bool, error)
	Put(digest *util.Digest, offset uint64, length int64, cursors Cursors) error
}

OffsetStore maps a digest to an offset within the data file. This is where the blob's contents may be found.

func NewCachingOffsetStore

func NewCachingOffsetStore(backend OffsetStore, size uint) OffsetStore

NewCachingOffsetStore is an adapter for OffsetStore that caches digests returned by/provided to previous calls of Get() and Put(). Cached entries are stored in a fixed-size hash table.

The purpose of this adapter is to significantly reduce the number of read operations on underlying storage. In the end it should reduce the running time of FindMissing() operations.

TODO(edsch): Should we add negative caching as well?

func NewDemultiplexingOffsetStore

func NewDemultiplexingOffsetStore(offsetStoreGetter OffsetStoreGetter) OffsetStore

NewDemultiplexingOffsetStore creates an OffsetStore that demultiplexes operations based on the instance name stored in the provided digest. This may be used for Action Cache purposes, where a single storage server may be used to store cached actions for multiple instance names.

func NewFileOffsetStore

func NewFileOffsetStore(storageType string, file ReadWriterAt, size uint64) OffsetStore

NewFileOffsetStore creates a file-based accessor for the offset store. The offset store maps a digest to an offset within the data file. This is where the blob's contents may be found.

Under the hood, this implementation uses a hash table with open addressing. In order to be self-cleaning, it uses a cuckoo-hash like approach, where objects may only be displaced to less preferential slots by objects with a higher offset. In other words, more recently stored blobs displace older ones.

type OffsetStoreGetter

type OffsetStoreGetter func(instanceName string) (OffsetStore, error)

OffsetStoreGetter is the callback type used by the demultiplexing offset store and is invoked for every operation.

type ReadWriterAt

type ReadWriterAt interface {
	io.ReaderAt
	io.WriterAt
}

ReadWriterAt is an interface for the file operations performed by the circular storage layer.

type StateStore

type StateStore interface {
	GetCursors() Cursors
	Allocate(sizeBytes int64) (uint64, error)
	Invalidate(offset uint64, sizeBytes int64) error
}

StateStore is where global metadata of the circular storage backend is stored, namely the read/write cursors where data is currently being stored in the data file.

func NewBulkAllocatingStateStore

func NewBulkAllocatingStateStore(stateStore StateStore, chunkSize uint64) StateStore

NewBulkAllocatingStateStore is an adapter for StateStore that reduces the number of Allocate() calls on the underlying implementation by allocating data as larger chunks. These chunks are then sub-allocated as needed.

func NewFileStateStore

func NewFileStateStore(file ReadWriterAt, dataSize uint64) (StateStore, error)

NewFileStateStore creates a new storage for global metadata of a circular storage backend. Right now only a set of read/write cursors are stored.

func NewPositiveSizedBlobStateStore

func NewPositiveSizedBlobStateStore(stateStore StateStore) StateStore

NewPositiveSizedBlobStateStore is an adapter for StateStore that forces allocations of blobs to be positive in size. Zero-sized allocations would permit multiple blobs to be stored at the same offset. Invalidations of such blobs would normally have no effect, causing the storage backend to be unable to repair itself.