blobstore

package
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 7, 2019 License: BSD-3-Clause Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MaxSize is the maximum blob size.
	MaxSize = 1 << 20
)

Variables

View Source 
var (
	// ErrBlobNotFound is returned when a blob cannot be found.
	ErrBlobNotFound = errors.New("blob not found")
	// ErrBlobTooBig is returned if a blob is larger than 1MB.
	ErrBlobTooBig = errors.New("blob size exceeded")
	// ErrBlobExists is returned if a blob exists when create is called.
	ErrBlobExists = errors.New("blob already exists")
	// ErrShuttingDown is returned when server is shutting down.
	ErrShuttingDown = errors.New("server is shutting down")
)
View Source 
var ErrTestIntentional = fmt.Errorf("intentional failure for test")

ErrTestIntentional is used for intentional test failures.

View Source 
var TestAlwaysFail = func(set, key string) error {
	return ErrTestIntentional
}

TestAlwaysFail will make all calls to blob store fail. ErrTestIntentional will be returned.

View Source 
var WithLazySaveOption = LazySaveOption{}

WithLazySaveOption provides an element to create lazySave parameters.

View Source 
var WithRetryOpt = RetryOpt{}

WithLazySaveOption provides an element to create lazySave parameters.

Functions

func TestFailAfterN 

func TestFailAfterN(n int) func(set, key string) error

TestFailAfterN will fail after N calls. ErrTestIntentional will be returned.

func TestFailPct 

func TestFailPct(n uint32, seed int64) func(set, key string) error

TestFailPct will fail the percentage of calls. ErrTestIntentional will be returned.

Types

type LazySaveOption 

type LazySaveOption struct{}

LazySaveOption provides access to LazySave Options that are methods on this struct.

func (LazySaveOption) Items 

func (l LazySaveOption) Items(limit, flushAt int) lazySaveOption

MaxItems will set the maximum number of items in queue, and when to begin flushing items, even though they haven't expired.

func (LazySaveOption) Logger 

func (l LazySaveOption) Logger(logger log.Adapter) lazySaveOption

Logger sets the logger for the async saver.

func (LazySaveOption) MaxTime 

func (l LazySaveOption) MaxTime(n time.Duration) lazySaveOption

MaxTime will set the maximum time an item will stay in cache before being flushed.

func (LazySaveOption) SaveTimeout 

func (l LazySaveOption) SaveTimeout(n time.Duration) lazySaveOption

SaveTimeout is the deadline set for saving items.

func (LazySaveOption) Savers 

func (l LazySaveOption) Savers(n int) lazySaveOption

Savers is the number of concurrent savers running.

func (LazySaveOption) Verbose 

func (l LazySaveOption) Verbose(b bool) lazySaveOption

Logger sets the logger for the async saver.

type LazySaver 

type LazySaver struct {
	// contains filtered or unexported fields
}

LazySaver enables lazy saving of blobs. Items pending saves can be modified and are instantly returned.

func NewLazySaver 

func NewLazySaver(store Store, opts ...lazySaveOption) (*LazySaver, error)

NewLazySaver will create a new lazy saver backed by the provided store.

func (*LazySaver) Delete 

func (l *LazySaver) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*LazySaver) Get 

func (l *LazySaver) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*LazySaver) Set 

func (l *LazySaver) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

func (*LazySaver) Shutdown 

func (l *LazySaver) Shutdown()

Shutdown should be called when the server is done writing and the remaining data should be flushed.

func (*LazySaver) Stats 

func (l *LazySaver) Stats() LazyStats

Stats returns stats.

type LazyStats 

type LazyStats struct {
	Options struct {
		MaxItems    int           `json:"max_items"`
		FlushItems  int           `json:"flush_items"`
		MaxTime     time.Duration `json:"max_time"`
		Savers      int           `json:"savers"`
		SaveTimeout time.Duration `json:"save_timeout"`
	} `json:"options"`
	ItemQueue        int     `json:"item_queue"`
	ItemBytes        int64   `json:"item_bytes"`
	Gets             int     `json:"gets"`
	Sets             int     `json:"sets"`
	Deletes          int     `json:"deletes"`
	LoadedBytes      int64   `json:"loaded_bytes"`
	SetBytes         int64   `json:"set_bytes"`
	GetBytes         int64   `json:"get_bytes"`
	OutGetBytes      int64   `json:"out_get_bytes"`
	OutSetBytes      int64   `json:"out_set_bytes"`
	OutGets          int     `json:"out_gets"`
	OutSets          int     `json:"out_sets"`
	OutDeletes       int     `json:"out_deletes"`
	SavedBytes       int64   `json:"saved_bytes"`
	GetSavedBytesPct float64 `json:"get_saved_pct"`
	SetSavedBytesPct float64 `json:"set_saved_pct"`
}

type MaxSizeStore 

type MaxSizeStore struct {
	// contains filtered or unexported fields
}

MaxSizeStore will make sure that no blobs are above the specified size. The upper storage limit is 65535 * maximum size.

func NewMaxSizeStore 

func NewMaxSizeStore(store Store, maxSize int) (*MaxSizeStore, error)

NewMaxSizeStore will split blobs into the specified size. Specify the upper byte limit. The upper storage limit is 65535 * maximum size.

func (*MaxSizeStore) Delete 

func (r *MaxSizeStore) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*MaxSizeStore) Get 

func (r *MaxSizeStore) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*MaxSizeStore) Set 

func (r *MaxSizeStore) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

type RetryOpt 

type RetryOpt struct{}

LazySaveOption provides access to LazySave Options that are methods on this struct.

func (RetryOpt) Backoff 

func (l RetryOpt) Backoff(get, set, del func() backoff.BackOff) RetryStoreOption

Backoff supplies individual backoff strategies for get, set and delete operations. If nil is supplied for any option, that strategy will not be updated.

func (RetryOpt) OpTimeout 

func (l RetryOpt) OpTimeout(duration time.Duration) RetryStoreOption

OpTimeout will set the context timeout for each operation attempt. Setting this to 0 will use parent context deadlines. Default is 1 second.

func (RetryOpt) PermanentErr 

func (l RetryOpt) PermanentErr(errs ...error) RetryStoreOption

PermanentErr will add errors to permanent error table.

type RetryStore 

type RetryStore struct {
	// contains filtered or unexported fields
}

RetryStore will apply retry functionality with customizable backoff.

func NewRetryStore 

func NewRetryStore(store Store, opts ...RetryStoreOption) (*RetryStore, error)

NewLazySaver will create a new lazy saver backed by the provided store.

func (*RetryStore) Delete 

func (r *RetryStore) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*RetryStore) Get 

func (r *RetryStore) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*RetryStore) Set 

func (r *RetryStore) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

type RetryStoreOption 

type RetryStoreOption func(*retryStoreOptions) error

type Store 

type Store interface {
	// GetBlob will retrieve a blob from Aerospike.
	// Expects data to have been stored with Set or Create.
	Get(ctx context.Context, set, key string) ([]byte, error)

	// SetTTL will store a byte blob in Aerospike with a given TTL.
	// Update policy is last write wins with no generation checks.
	// To retrieve a blob use GetBlob with same set/key values.
	// If ttl is 0 no expiration will be set.
	// A blob may not be bigger than 1MB.
	Set(ctx context.Context, set, key string, val []byte) error

	// Delete will delete a blob.
	// If the blob could not be deleted an error is returned.
	// If the blob does not exist, the request has no effect.
	Delete(ctx context.Context, set, key string) error
}

Store supplies a blob storage interface.

type TestStorage 

type TestStorage struct {
	GetCallback func(set, key string)

	SetCallback func(set, key string, b []byte)

	DeleteCallback func(set, key string)
	// contains filtered or unexported fields
}

TestStorage provides callbacks and interceptors for easy testing of blob store failures.

func NewTestStore 

func NewTestStore(store Store) *TestStorage

NewTestStore is a store that can be used for testing. Without modifications it allows

func (*TestStorage) Delete 

func (t *TestStorage) Delete(ctx context.Context, set, key string) error

Delete a blob.

func (*TestStorage) FailDelete 

func (t *TestStorage) FailDelete(check func(set, key string) error)

FailDelete will call the provided function. If nil is returned, the original store is queried, otherwise the error is returned.

func (*TestStorage) FailGet 

func (t *TestStorage) FailGet(check func(set, key string) error)

FailGetKey will call the provided function. If nil is returned, the original store is queried, otherwise the error is returned.

func (*TestStorage) FailSet 

func (t *TestStorage) FailSet(check func(set, key string) error)

FailSet will call the provided function. If nil is returned, the original store is queried, otherwise the error is returned.

func (*TestStorage) Get 

func (t *TestStorage) Get(ctx context.Context, set, key string) ([]byte, error)

Get a blob.

func (*TestStorage) Set 

func (t *TestStorage) Set(ctx context.Context, set, key string, b []byte) error

Set a blob.

type WithSet 

type WithSet interface {
	// GetBlob will retrieve a blob from Aerospike.
	// Expects data to have been stored with Set or Create.
	Get(ctx context.Context, key string) ([]byte, error)

	// SetTTL will store a byte blob in Aerospike with a given TTL.
	// Update policy is last write wins with no generation checks.
	// To retrieve a blob use GetBlob with same set/key values.
	// If ttl is 0 no expiration will be set.
	// A blob may not be bigger than 1MB.
	Set(ctx context.Context, key string, val []byte) error

	// Delete will delete a blob.
	// If the blob could not be deleted an error is returned.
	// If the blob does not exist, the request has no effect.
	Delete(ctx context.Context, key string) error

	// Store returns the underlying store.
	Store() Store
}

func StoreWithSet 

func StoreWithSet(s Store, set string) WithSet

StoreWithSet wraps storage with a specific set. The returned interface is safe for concurrent use.

Source Files

View all Source files

Directories

Path Synopsis
Package bstest supplies helpers to test blobstores.
 Package bstest supplies helpers to test blobstores.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.