refstore

Documentation

Index

• Variables
• func AvailableAdapters() (list []string)
• func RegisterAdapter(name string, adapter AdapterConstructor)
• type AccessList
  • func NewAccessList(maxItems, totalSizeQuota int) (*AccessList, error)
  • func (a *AccessList) Add(f *FileRef) (evicted []*FileRef)
  • func (a *AccessList) Peek(id Identifier) *FileRef
  • func (a *AccessList) Prime(init []*FileRef) (evicted []*FileRef)
  • func (a *AccessList) Touch(id Identifier)
• type Adapter
  • func NewStore(dsn string, rp RetentionPolicy) (Adapter, error)
• type AdapterConstructor
• type ErrInvalidIdentifier
  • func (err *ErrInvalidIdentifier) Error() string
  • func (err *ErrInvalidIdentifier) Unwrap() error
• type ErrStoreAlreadyTaken
  • func (err ErrStoreAlreadyTaken) Error() string
• type FileRef
• type Identifier
  • func NewIdentifier(contents []byte) Identifier
  • func ParseIdentifier(b []byte) (Identifier, error)
  • func ReadIdentifier(r io.Reader) (Identifier, error)
  • func ToIdentifier(b []byte) (Identifier, error)
  • func (id Identifier) Raw() string
  • func (id Identifier) String() string
• type KeepForever
  • func (*KeepForever) Add(f *FileRef) []*FileRef
  • func (*KeepForever) Peek(Identifier) *FileRef
  • func (*KeepForever) Prime([]*FileRef) []*FileRef
  • func (*KeepForever) Touch(Identifier)
• type PurgeOnStart
  • func (*PurgeOnStart) Add(f *FileRef) []*FileRef
  • func (*PurgeOnStart) Peek(Identifier) *FileRef
  • func (*PurgeOnStart) Prime(f []*FileRef) []*FileRef
  • func (*PurgeOnStart) Touch(Identifier)
• type RetentionPolicy

Variables

var ErrInvalidAccessListConfig = errors.New("invalid access list configuration, max. item count and file size can't both be infinite")

ErrInvalidAccessListConfig is returned from NewAccessList when the quota arguments (maxItems and totalSizeQuota) are both <= 0.

var ErrUnknownReference = errors.New("unknown reference")

ErrUnknownReference can be returned from Adapter implementations, if a given Identifier is unknown to them.

Functions

func AvailableAdapters

func AvailableAdapters() (list []string)

func RegisterAdapter

func RegisterAdapter(name string, adapter AdapterConstructor)

RegisterAdapter will remember the given adapter with under the given name. It will panic, if the name is already taken.

Types

type AccessList

type AccessList struct {
	// contains filtered or unexported fields
}

AccessList implements a size- and/or space-limited RetentionPolicy. Size-limited means, that the AccessList may just accept a limited number of entries, while the space limitation constrains the total file size of all entries.

Reference file usage (through Touch and Add) moves entries to the front of the access list, while adding entries may evict entries from the back when limits are exceeded. In doing so, AccessList acts as cache with LRU semantics.

func NewAccessList

func NewAccessList(maxItems, totalSizeQuota int) (*AccessList, error)

func (*AccessList) Add

func (a *AccessList) Add(f *FileRef) (evicted []*FileRef)

Add implements the RetentionPolicy interface.

func (*AccessList) Peek

func (a *AccessList) Peek(id Identifier) *FileRef

Peek implements the RetentionPolicy interface.

func (*AccessList) Prime

func (a *AccessList) Prime(init []*FileRef) (evicted []*FileRef)

Prime implements the RetentionPolicy interface.

func (*AccessList) Touch

func (a *AccessList) Touch(id Identifier)

Touch implements the RetentionPolicy interface.

type Adapter

type Adapter interface {
	// CopyFile copies a file with the given ID to the target path. It may
	// return ErrUnknownReference if the ID is unknown.
	CopyFile(log *zap.Logger, id Identifier, w io.Writer) error

	// Store saves the content in the adapter backend.
	Store(log *zap.Logger, r io.Reader) error

	// Exists checks whether the given reference identifier exists in this
	// storage adapter.
	Exists(id Identifier) bool
}

The Adapter interface describes the protocol to interact with different storage backends.

func NewStore

func NewStore(dsn string, rp RetentionPolicy) (Adapter, error)

NewStore creates a new reference store with the given DSN and retention policy. The adapter name is extracted from the DSN, i.e. the dir adapter requires a DSN of the form "dir://".

If rp is nil, a "keep forever" policy is assumed and an instance of KeepForever is passed to the reference store constructor.

type AdapterConstructor

type AdapterConstructor func(*url.URL, RetentionPolicy) (Adapter, error)

AdapterConstructor constructs a new adapter implementation from the given config DSN and retention policy.

type ErrInvalidIdentifier

type ErrInvalidIdentifier struct {
	// contains filtered or unexported fields
}

func (*ErrInvalidIdentifier) Error

func (err *ErrInvalidIdentifier) Error() string

func (*ErrInvalidIdentifier) Unwrap

func (err *ErrInvalidIdentifier) Unwrap() error

type ErrStoreAlreadyTaken

type ErrStoreAlreadyTaken string

func (ErrStoreAlreadyTaken) Error

func (err ErrStoreAlreadyTaken) Error() string

type FileRef

type FileRef struct {
	ID   Identifier
	Size int
}

FileRef represents a reference file. It does not contain any file data, only enough meta data to find the file content in a reference store (by its ID), and make decisions about evictions (using its Size).

type Identifier

type Identifier string

File references are identified by their checksum.

func NewIdentifier

func NewIdentifier(contents []byte) Identifier

NewIdentifier calculates the reference ID from the given file contents.

func ParseIdentifier

func ParseIdentifier(b []byte) (Identifier, error)

ParseIdentifier takes an input in the form "sha256:...." and transforms it into an Identifier. Parsing errors are reported as ErrInvalidIdentifier.

func ReadIdentifier

func ReadIdentifier(r io.Reader) (Identifier, error)

ReadIdentifier creates an identifier of the contents read from r.

func ToIdentifier

func ToIdentifier(b []byte) (Identifier, error)

ToIdentifier converts b into the canonical Identifier representation. This returns an error if len(b) is not sha256.Size (32).

func (Identifier) Raw

func (id Identifier) Raw() string

Raw returns the raw URL-safe Base64-encoded Identifier, i.e. id.String(), but without the "sha256:" prefix. For now, this is identical to string(id).

func (Identifier) String

func (id Identifier) String() string

String returns the canonical string representation of the Identifier, i.e. something like "sha256:<bas64 chars>".

type KeepForever

type KeepForever struct{}

KeepForever is a no-op retention policy, i.e. reference stores should keeps all files forever.

func (*KeepForever) Add

func (*KeepForever) Add(f *FileRef) []*FileRef

func (*KeepForever) Peek

func (*KeepForever) Peek(Identifier) *FileRef

func (*KeepForever) Prime

func (*KeepForever) Prime([]*FileRef) []*FileRef

func (*KeepForever) Touch

func (*KeepForever) Touch(Identifier)

type PurgeOnStart

type PurgeOnStart struct{}

PurgeOnStart will delete all files when the reference store is instantiated. This typically happens once on service startup.

func (*PurgeOnStart) Add

func (*PurgeOnStart) Add(f *FileRef) []*FileRef

func (*PurgeOnStart) Peek

func (*PurgeOnStart) Peek(Identifier) *FileRef

func (*PurgeOnStart) Prime

func (*PurgeOnStart) Prime(f []*FileRef) []*FileRef

func (*PurgeOnStart) Touch

func (*PurgeOnStart) Touch(Identifier)

type RetentionPolicy

type RetentionPolicy interface {
	// Prime should be used to fill the internal access list on startup.
	// It returns the list of evicted file references (which the reference
	// store should then delete).
	//
	// Semantically, Prime is equivalent to calling Add multiple times:
	//
	//	// add multiple references at once
	//	Prime([]*FileRef{a, b, c})
	//	// add references individually
	//	Add(a); Add(b); Add(c)
	//
	// (Note: this example omits handling of evicted references.)
	Prime(refs []*FileRef) (evicted []*FileRef)

	// Touch marks an identifier as recently used.
	Touch(id Identifier)

	// Add will insert a file reference into an internal store and return
	// file references which should be evicted. Depending on the chosen
	// implementation, the file argument can be part of the returned pruning
	// set.
	Add(ref *FileRef) (evicted []*FileRef)

	// Peek performa a lookup of the given identifier and returns a copy
	// returns a copy of the reference file, should it exist. Otherwise
	// this simply returns nil. In contract to Add and Touch, Peek should
	// not have any side effects.
	//
	// Note: Peek is intended to help in testing. Calling Peek is safe to
	// use concurrently, but it makes no guarantees as to whether the
	// returned value still exists in the access list after the next call
	// to Add().
	Peek(id Identifier) (existing *FileRef)
}

A RetentionPolicy describes how various reference store implementations should handle deletion of old references.

Note, that the retention policy only keeps track of Identifiers, not of the physical storage location. Reference store implementation need to properly act on the return values of Prime and Add, and remove the referenced files.