cas

Documentation

Overview

pakcage cas provides a content-addressable storage interface on top of an gocloud compatible interface.

The implementation is written in such way that any service that provides an object storage (s3-like apis) can be used.

MinIO is recommended for instalations that don't rely on a cloud provider

Index

• Constants
• func RefCalculator(out *Ref, content io.Reader) io.ReadCloser
• type C
  • func Open(ctx context.Context, newBucket NewTable) (*C, error)
  • func (c *C) Close() error
  • func (c *C) Exists(ctx context.Context, ref Ref) (bool, error)
  • func (c *C) GetContent(ctx context.Context, w io.Writer, ref Ref) error
  • func (c *C) PutContent(ctx context.Context, content io.Reader) (Ref, error)
• type Err
  • func (e Err) Error() string
• type KV
• type Mover
• type NewTable
• type Ref
  • func PrecomputeHashBytes(buf []byte) (ref Ref)
  • func (r Ref) HexPath(n int) string
  • func (r Ref) String() string
• type RollingRef
  • func NewRollingRef() RollingRef

Constants

const (
	ErrNotFound = Err("cas reference could not be found")
)

Functions

func RefCalculator

func RefCalculator(out *Ref, content io.Reader) io.ReadCloser

RefCaclulator returns a reader that computes the hash from the given content as consumers read data.

The Ref *pointer is updated when content.Read returns 0 bytes or an error is found, including io.EOF

Types

type C

type C struct {
	// contains filtered or unexported fields
}

C implements the CAS abstraction on top of a S3 compatible storage

func Open

func Open(ctx context.Context, newBucket NewTable) (*C, error)

Open a new CAS store using newBucket to acquire the remote item

func (*C) Close

func (c *C) Close() error

Close the underlying bucket

func (*C) Exists

func (c *C) Exists(ctx context.Context, ref Ref) (bool, error)

Exists returns true if the ref already exists

func (*C) GetContent

func (c *C) GetContent(ctx context.Context, w io.Writer, ref Ref) error

Get writes the object at ref to the given output it returns the underlying KV error without any modification

func (*C) PutContent

func (c *C) PutContent(ctx context.Context, content io.Reader) (Ref, error)

PutContent writes content to a temporary object and later copies that object to the final path under the sha256 hash.

This avoids reading the object twice but might incur in costs on S3-like services, even though the temporary object is alive for a short period of time.

This function does not perform deduplication, so if the content already exists, it will be re-uploaded. The Copy/Move operation wont be executed in this scenario.

If the provided KV object implementes the Mover interface, then instead of Copy/Delete cas will use the Move operation.

type Err

type Err string

Err represents errors that don't carry any extra information

func (Err) Error

func (e Err) Error() string

type KV

type KV interface {
	io.Closer
	Write(context.Context, string, io.Reader) (int64, error)
	Read(context.Context, io.Writer, string) (int64, error)
	Copy(context.Context, string, string) error
	Delete(context.Context, string) error
	Exists(context.Context, string) (bool, error)
}

KV represents the bare minimum abstraction required by cas

type Mover

type Mover interface {
	Move(context.Context, string, string) error
}

MovableObjects contains the methods required by cas which help reduce the load on the remote server

type NewTable

type NewTable func(context.Context) (KV, error)

NewTable should return a KV object which is used to store the items

cas package is responsible for closing the KV

type Ref

type Ref [32]byte

Ref contains the binary value of the sha256 hash which identifies any object

func PrecomputeHashBytes

func PrecomputeHashBytes(buf []byte) (ref Ref)

PrecomputeHashBytes returns the expected Ref value for the given set of bytes

func (Ref) HexPath

func (r Ref) HexPath(n int) string

Returns the hex encoded path with the first hex-bytes used as directories

n MUST be less than 32

func (Ref) String

func (r Ref) String() string

String returns the hex encoding of this object

type RollingRef

type RollingRef interface {
	io.WriteCloser
	io.ByteWriter
	Ref() Ref
	Reset()
}

RollingRef computes a ref value as bytes are written to it it is just a wraper over a hash.Hash

func NewRollingRef

func NewRollingRef() RollingRef

NewRollingRef configures a new rolling hash object