blobstore

package
v1.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 30, 2021 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// MaxSize is the maximum blob size.
	MaxSize = 1 << 20
)

Variables

View Source
var (
	// ErrBlobNotFound is returned when a blob cannot be found.
	ErrBlobNotFound = errors.New("blob not found")
	// ErrBlobTooBig is returned if a blob is larger than 1MB.
	ErrBlobTooBig = errors.New("blob size exceeded")
	// ErrBlobExists is returned if a blob exists when create is called.
	ErrBlobExists = errors.New("blob already exists")
	// ErrShuttingDown is returned when server is shutting down.
	ErrShuttingDown = errors.New("server is shutting down")
)
View Source
var ErrTestIntentional = fmt.Errorf("intentional failure for test")

ErrTestIntentional is used for intentional test failures.

View Source
var TestAlwaysFail = func(set, key string) error {
	return ErrTestIntentional
}

TestAlwaysFail will make all calls to blob store fail. ErrTestIntentional will be returned.

View Source
var WithLazySaveOption = LazySaveOption{}

WithLazySaveOption provides an element to create lazySave parameters.

View Source
var WithRetryOpt = RetryOpt{}

WithLazySaveOption provides an element to create lazySave parameters.

Functions

func TestFailAfterN

func TestFailAfterN(n int) func(set, key string) error

TestFailAfterN will fail after N calls. ErrTestIntentional will be returned.

func TestFailPct

func TestFailPct(n uint32, seed int64) func(set, key string) error

TestFailPct will fail the percentage of calls. ErrTestIntentional will be returned.

Types

type LazySaveOption

type LazySaveOption struct{}

LazySaveOption provides access to LazySave Options that are methods on this struct.

func (LazySaveOption) Items

func (l LazySaveOption) Items(limit, flushAt int) lazySaveOption

MaxItems will set the maximum number of items in queue, and when to begin flushing items, even though they haven't expired.

func (LazySaveOption) Logger

func (l LazySaveOption) Logger(logger log.Adapter) lazySaveOption

Logger sets the logger for the async saver.

func (LazySaveOption) MaxTime

func (l LazySaveOption) MaxTime(n time.Duration) lazySaveOption

MaxTime will set the maximum time an item will stay in cache before being flushed.

func (LazySaveOption) SaveTimeout

func (l LazySaveOption) SaveTimeout(n time.Duration) lazySaveOption

SaveTimeout is the deadline set for saving items.

func (LazySaveOption) Savers

func (l LazySaveOption) Savers(n int) lazySaveOption

Savers is the number of concurrent savers running.

func (LazySaveOption) Verbose

func (l LazySaveOption) Verbose(b bool) lazySaveOption

Logger sets the logger for the async saver.

type LazySaver

type LazySaver struct {
	// contains filtered or unexported fields
}

LazySaver enables lazy saving of blobs. Items pending saves can be modified and are instantly returned.

func NewLazySaver

func NewLazySaver(store Store, opts ...lazySaveOption) (*LazySaver, error)

NewLazySaver will create a new lazy saver backed by the provided store.

func (*LazySaver) Delete

func (l *LazySaver) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*LazySaver) Get

func (l *LazySaver) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*LazySaver) Set

func (l *LazySaver) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

func (*LazySaver) Shutdown

func (l *LazySaver) Shutdown()

Shutdown should be called when the server is done writing and the remaining data should be flushed.

func (*LazySaver) Stats

func (l *LazySaver) Stats() LazyStats

Stats returns stats.

type LazyStats

type LazyStats struct {
	Options struct {
		MaxItems    int           `json:"max_items"`
		FlushItems  int           `json:"flush_items"`
		MaxTime     time.Duration `json:"max_time"`
		Savers      int           `json:"savers"`
		SaveTimeout time.Duration `json:"save_timeout"`
	} `json:"options"`
	ItemQueue        int     `json:"item_queue"`
	ItemBytes        int64   `json:"item_bytes"`
	Gets             int     `json:"gets"`
	Sets             int     `json:"sets"`
	Deletes          int     `json:"deletes"`
	LoadedBytes      int64   `json:"loaded_bytes"`
	SetBytes         int64   `json:"set_bytes"`
	GetBytes         int64   `json:"get_bytes"`
	OutGetBytes      int64   `json:"out_get_bytes"`
	OutSetBytes      int64   `json:"out_set_bytes"`
	OutGets          int     `json:"out_gets"`
	OutSets          int     `json:"out_sets"`
	OutDeletes       int     `json:"out_deletes"`
	SavedBytes       int64   `json:"saved_bytes"`
	GetSavedBytesPct float64 `json:"get_saved_pct"`
	SetSavedBytesPct float64 `json:"set_saved_pct"`
}

type MaxSizeStore

type MaxSizeStore struct {
	// contains filtered or unexported fields
}

MaxSizeStore will make sure that no blobs are above the specified size. The upper storage limit is 65535 * maximum size.

func NewMaxSizeStore

func NewMaxSizeStore(store Store, maxSize int) (*MaxSizeStore, error)

NewMaxSizeStore will split blobs into the specified size. Specify the upper byte limit. The upper storage limit is 65535 * maximum size.

func (*MaxSizeStore) Delete

func (r *MaxSizeStore) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*MaxSizeStore) Get

func (r *MaxSizeStore) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*MaxSizeStore) Set

func (r *MaxSizeStore) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

type RetryOpt

type RetryOpt struct{}

LazySaveOption provides access to LazySave Options that are methods on this struct.

func (RetryOpt) Backoff

func (l RetryOpt) Backoff(get, set, del func() backoff.BackOff) RetryStoreOption

Backoff supplies individual backoff strategies for get, set and delete operations. If nil is supplied for any option, that strategy will not be updated.

func (RetryOpt) OpTimeout

func (l RetryOpt) OpTimeout(duration time.Duration) RetryStoreOption

OpTimeout will set the context timeout for each operation attempt. Setting this to 0 will use parent context deadlines. Default is 1 second.

func (RetryOpt) PermanentErr

func (l RetryOpt) PermanentErr(errs ...error) RetryStoreOption

PermanentErr will add errors to permanent error table.

type RetryStore

type RetryStore struct {
	// contains filtered or unexported fields
}

RetryStore will apply retry functionality with customizable backoff.

func NewRetryStore

func NewRetryStore(store Store, opts ...RetryStoreOption) (*RetryStore, error)

NewLazySaver will create a new lazy saver backed by the provided store.

func (*RetryStore) Delete

func (r *RetryStore) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*RetryStore) Get

func (r *RetryStore) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*RetryStore) Set

func (r *RetryStore) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

type RetryStoreOption

type RetryStoreOption func(*retryStoreOptions) error

type Store

type Store interface {
	// GetBlob will retrieve a blob from Aerospike.
	// Expects data to have been stored with Set or Create.
	Get(ctx context.Context, set, key string) ([]byte, error)

	// SetTTL will store a byte blob in Aerospike with a given TTL.
	// Update policy is last write wins with no generation checks.
	// To retrieve a blob use GetBlob with same set/key values.
	// If ttl is 0 no expiration will be set.
	// A blob may not be bigger than 1MB.
	Set(ctx context.Context, set, key string, val []byte) error

	// Delete will delete a blob.
	// If the blob could not be deleted an error is returned.
	// If the blob does not exist, the request has no effect.
	Delete(ctx context.Context, set, key string) error
}

Store supplies a blob storage interface.

type TestStorage

type TestStorage struct {
	GetCallback func(set, key string)

	SetCallback func(set, key string, b []byte)

	DeleteCallback func(set, key string)
	// contains filtered or unexported fields
}

TestStorage provides callbacks and interceptors for easy testing of blob store failures.

func NewTestStore

func NewTestStore(store Store) *TestStorage

NewTestStore is a store that can be used for testing. Without modifications it allows

func (*TestStorage) Delete

func (t *TestStorage) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*TestStorage) FailDelete

func (t *TestStorage) FailDelete(check func(set, key string) error)

FailDelete will call the provided function. If nil is returned, the original store is queried, otherwise the error is returned.

func (*TestStorage) FailGet

func (t *TestStorage) FailGet(check func(set, key string) error)

FailGetKey will call the provided function. If nil is returned, the original store is queried, otherwise the error is returned.

func (*TestStorage) FailSet

func (t *TestStorage) FailSet(check func(set, key string) error)

FailSet will call the provided function. If nil is returned, the original store is queried, otherwise the error is returned.

func (*TestStorage) Get

func (t *TestStorage) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*TestStorage) Set

func (t *TestStorage) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

type WithSet

type WithSet interface {
	// GetBlob will retrieve a blob from Aerospike.
	// Expects data to have been stored with Set or Create.
	Get(ctx context.Context, key string) ([]byte, error)

	// SetTTL will store a byte blob in Aerospike with a given TTL.
	// Update policy is last write wins with no generation checks.
	// To retrieve a blob use GetBlob with same set/key values.
	// If ttl is 0 no expiration will be set.
	// A blob may not be bigger than 1MB.
	Set(ctx context.Context, key string, val []byte) error

	// Delete will delete a blob.
	// If the blob could not be deleted an error is returned.
	// If the blob does not exist, the request has no effect.
	Delete(ctx context.Context, key string) error

	// Store returns the underlying store.
	Store() Store
}

func StoreWithSet

func StoreWithSet(s Store, set string) WithSet

StoreWithSet wraps storage with a specific set. The returned interface is safe for concurrent use.

Directories

Path Synopsis
Package bstest supplies helpers to test blobstores.
Package bstest supplies helpers to test blobstores.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL