sharded

Documentation

Overview

Package sharded implements common support for sharded blob providers, such as filesystem or webdav.

Package sharded implements common support for sharded blob providers, such as filesystem or webdav.

Index

• Constants
• type Impl
• type Parameters
  • func DefaultParameters(shards []int) *Parameters
  • func (p *Parameters) Clone() *Parameters
  • func (p *Parameters) GetShardDirectoryAndBlob(rootPath string, blobID blob.ID) (string, blob.ID)
  • func (p *Parameters) Load(r io.Reader) error
  • func (p *Parameters) Save(w io.Writer) error
• type PrefixAndShards
• type Storage
  • func (s *Storage) DeleteBlob(ctx context.Context, blobID blob.ID) error
  • func (s *Storage) GetBlob(ctx context.Context, blobID blob.ID, offset, length int64, ...) error
  • func (s *Storage) GetMetadata(ctx context.Context, blobID blob.ID) (blob.Metadata, error)
  • func (s *Storage) GetShardedPathAndFilePath(ctx context.Context, blobID blob.ID) (shardPath, filePath string, err error)
  • func (s *Storage) ListBlobs(ctx context.Context, prefix blob.ID, callback func(blob.Metadata) error) error
  • func (s *Storage) PutBlob(ctx context.Context, blobID blob.ID, data blob.Bytes) error
  • func (s *Storage) SetTime(ctx context.Context, blobID blob.ID, n time.Time) error

Constants

const CompleteBlobSuffix = ".f"

CompleteBlobSuffix is the extension for sharded blobs that have completed writing.

const ParametersFile = ".shards"

ParametersFile is the name of the hidden parameters file in a sharded storage.

Types

type Impl

type Impl interface {
	GetBlobFromPath(ctx context.Context, dirPath, filePath string, offset, length int64, output *gather.WriteBuffer) error
	GetMetadataFromPath(ctx context.Context, dirPath, filePath string) (blob.Metadata, error)
	PutBlobInPath(ctx context.Context, dirPath, filePath string, dataSlices blob.Bytes) error
	SetTimeInPath(ctx context.Context, dirPath, filePath string, t time.Time) error
	DeleteBlobInPath(ctx context.Context, dirPath, filePath string) error
	ReadDir(ctx context.Context, path string) ([]os.FileInfo, error)
}

Impl must be implemented by underlying provided.

type Parameters

type Parameters struct {
	DefaultShards   []int             `json:"default"`
	UnshardedLength int               `json:"maxNonShardedLength"`
	Overrides       []PrefixAndShards `json:"overrides,omitempty"`
}

Parameters contains sharded storage configuration optionally persisted in the storage itself.

func DefaultParameters

func DefaultParameters(shards []int) *Parameters

DefaultParameters constructs Parameters based on the provided shards specification.

func (*Parameters) Clone

func (p *Parameters) Clone() *Parameters

Clone returns a clone of sharding parameters.

func (*Parameters) GetShardDirectoryAndBlob

func (p *Parameters) GetShardDirectoryAndBlob(rootPath string, blobID blob.ID) (string, blob.ID)

GetShardDirectoryAndBlob gets that sharded directory and blob ID for a provided blob.

func (*Parameters) Load

func (p *Parameters) Load(r io.Reader) error

Load loads the Parameters from the provided reader.

func (*Parameters) Save

func (p *Parameters) Save(w io.Writer) error

Save saves the parameters to the provided writer.

type PrefixAndShards

type PrefixAndShards struct {
	Prefix blob.ID `json:"prefix"`
	Shards []int   `json:"shards"`
}

PrefixAndShards defines shards to use for a particular blob ID prefix.

type Storage

type Storage struct {
	Impl Impl

	RootPath        string
	Shards          []int
	ListParallelism int
	// contains filtered or unexported fields
}

Storage provides common implementation of sharded storage.

func (*Storage) DeleteBlob

func (s *Storage) DeleteBlob(ctx context.Context, blobID blob.ID) error

DeleteBlob implements blob.Storage.

func (*Storage) GetBlob

func (s *Storage) GetBlob(ctx context.Context, blobID blob.ID, offset, length int64, output *gather.WriteBuffer) error

GetBlob implements blob.Storage.

func (*Storage) GetMetadata

func (s *Storage) GetMetadata(ctx context.Context, blobID blob.ID) (blob.Metadata, error)

GetMetadata implements blob.Storage.

func (*Storage) GetShardedPathAndFilePath

func (s *Storage) GetShardedPathAndFilePath(ctx context.Context, blobID blob.ID) (shardPath, filePath string, err error)

GetShardedPathAndFilePath returns the path of the shard and file name within the shard for a given blob ID.

func (*Storage) ListBlobs

func (s *Storage) ListBlobs(ctx context.Context, prefix blob.ID, callback func(blob.Metadata) error) error

ListBlobs implements blob.Storage.

func (*Storage) PutBlob

func (s *Storage) PutBlob(ctx context.Context, blobID blob.ID, data blob.Bytes) error

PutBlob implements blob.Storage.

func (*Storage) SetTime

func (s *Storage) SetTime(ctx context.Context, blobID blob.ID, n time.Time) error

SetTime implements blob.Storage.