sharky

Documentation

Index

• Constants
• Variables
• type Location
  • func LocationFromBinary(buf []byte) (Location, error)
  • func (l *Location) MarshalBinary() ([]byte, error)
  • func (l *Location) UnmarshalBinary(buf []byte) error
• type Recovery
  • func NewRecovery(dir string, shardCnt int, datasize int) (*Recovery, error)
  • func (r *Recovery) Add(loc Location) error
  • func (r *Recovery) Close() error
  • func (r *Recovery) Read(ctx context.Context, loc Location, buf []byte) error
  • func (r *Recovery) Save() error
• type Store
  • func New(basedir fs.FS, shardCnt int, maxDataSize int) (*Store, error)
  • func (s *Store) Close() error
  • func (s *Store) Metrics() []prometheus.Collector
  • func (s *Store) Read(ctx context.Context, loc Location, buf []byte) (err error)
  • func (s *Store) Release(ctx context.Context, loc Location) error
  • func (s *Store) Write(ctx context.Context, data []byte) (loc Location, err error)

Constants

const LocationSize int = 7

LocationSize is the size of the byte representation of Location

Variables

var (
	// ErrTooLong returned by Write if the blob length exceeds the max blobsize.
	ErrTooLong = errors.New("data too long")
	// ErrQuitting returned by Write when the store is Closed before the write completes.
	ErrQuitting = errors.New("quitting")
)

var ErrShardNotFound = errors.New("shard not found")

Types

type Location

type Location struct {
	Shard  uint8
	Slot   uint32
	Length uint16
}

Location models the location <shard, slot, length> of a chunk

func LocationFromBinary

func LocationFromBinary(buf []byte) (Location, error)

LocationFromBinary is a helper to construct a Location object from byte representation

func (*Location) MarshalBinary

func (l *Location) MarshalBinary() ([]byte, error)

MarshalBinary returns byte representation of location

func (*Location) UnmarshalBinary

func (l *Location) UnmarshalBinary(buf []byte) error

UnmarshalBinary constructs the location from byte representation

type Recovery

type Recovery struct {
	// contains filtered or unexported fields
}

Recovery allows disaster recovery.

func NewRecovery

func NewRecovery(dir string, shardCnt int, datasize int) (*Recovery, error)

func (*Recovery) Add

func (r *Recovery) Add(loc Location) error

Add marks a location as used (not free).

func (*Recovery) Close

func (r *Recovery) Close() error

Close closes data and free slots files of the recovery (without saving).

func (*Recovery) Read

func (r *Recovery) Read(ctx context.Context, loc Location, buf []byte) error

func (*Recovery) Save

func (r *Recovery) Save() error

Save saves all free slots files of the recovery (without closing).

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store models the sharded fix-length blobstore Design provides lockless sharding: - shard choice responding to backpressure by running operation - read prioritisation over writing - free slots allow write

func New

func New(basedir fs.FS, shardCnt int, maxDataSize int) (*Store, error)

New constructs a sharded blobstore arguments: - base directory string - shard count - positive integer < 256 - cannot be zero or expect panic - shard size - positive integer multiple of 8 - for others expect undefined behaviour - maxDataSize - positive integer representing the maximum blob size to be stored

func (*Store) Close

func (s *Store) Close() error

Close closes each shard and return incidental errors from each shard

func (*Store) Metrics

func (s *Store) Metrics() []prometheus.Collector

Metrics returns set of prometheus collectors.

func (*Store) Read

func (s *Store) Read(ctx context.Context, loc Location, buf []byte) (err error)

Read reads the content of the blob found at location into the byte buffer given The location is assumed to be obtained by an earlier Write call storing the blob

func (*Store) Release

func (s *Store) Release(ctx context.Context, loc Location) error

Release gives back the slot to the shard From here on the slot can be reused and overwritten Release is meant to be called when an entry in the upstream db is removed Note that releasing is not safe for obfuscating earlier content, since even after reuse, the slot may be used by a very short blob and leaves the rest of the old blob bytes untouched

func (*Store) Write

func (s *Store) Write(ctx context.Context, data []byte) (loc Location, err error)

Write stores a new blob and returns its location to be used as a reference It can be given to a Read call to return the stored blob.