storages

README

Collection of storages (and wrappers)

Different implementations of storages with same abstract interface:

// Thread-safe storage for key-value
type Storage interface {
	// Put single item to storage. If already exists - override
	Put(key []byte, data []byte) error
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
	// Delete key and value
	Del(key []byte) error
	// Iterate over all keys. Modification during iteration may cause undefined behaviour (mostly - dead-lock)
	Keys(handler func(key []byte) error) error
    // Close storage if needs
    io.Closer
}

See documentation for details

one more example...

You can create different storage types just by URL (if you imported required package):

For example, use Redis db as a backend

storage, err := std.Create("redis://localhost")
if err != nil {
    panic(err)
}
defer storage.Close()

CLI tools

Binary

Look to releases section

Debian/Ubuntu

Add public Bintray key

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61

Add repository

• supported distribution: trusty, xenial, bionic, buster, wheezy

echo "deb https://dl.bintray.com/reddec/storages-debian {distribution} main" | sudo tee -a /etc/apt/sources.list

Ubuntu 18.04 (bionic)

echo "deb https://dl.bintray.com/reddec/storages-debian bionic main" | sudo tee -a /etc/apt/sources.list

Ubuntu 16.04 (xenial)

echo "deb https://dl.bintray.com/reddec/storages-debian xenial main" | sudo tee -a /etc/apt/sources.list

Update cache

sudo apt-get update

Install

sudo apt-get install storages

Build from source

• requires Go 1.13+

go get github.com/reddec/storages/cmd/...

License

The wrappers itself licensed under MIT but used libraries may have different license politics.

Documentation

Index

• func AllKeys(storage Storage) ([][]byte, error)
• func AllKeysString(storage Storage) ([]string, error)
• func AllNamespaces(storage NamespacedStorage) ([][]byte, error)
• func AllNamespacesString(storage NamespacedStorage) ([]string, error)
• func Redundant(writer DWriter, reader DReader, keysDeduplication Dedup, back ...Storage) *redundant
• func RedundantAll(keysDeduplication Dedup, back ...Storage) *redundant
• type Accessor
• type BatchedStorage
• type Clearable
• type DReader
  • func First() DReader
• type DWriter
  • func Any() DWriter
  • func AtLeast(minWrite int) DWriter
• type Dedup
• type Iterator
• type KV
• type LegacyQueue
• type LimitedQueue
• type NamespacedStorage
• type Queue
• type Reader
• type ShardPool
• type Storage
  • func Compressed(storage Storage) Storage
  • func Sharded(pool ShardPool) Storage
• type Writer

Functions

func AllKeys

func AllKeys(storage Storage) ([][]byte, error)

Extract all keys from storage as-is

func AllKeysString

func AllKeysString(storage Storage) ([]string, error)

Extract all keys from storage and convert it to string

func AllNamespaces

func AllNamespaces(storage NamespacedStorage) ([][]byte, error)

Extract all namespaces from storage as-is

func AllNamespacesString

func AllNamespacesString(storage NamespacedStorage) ([]string, error)

Extract all namespaces from storage as string

func Redundant

func Redundant(writer DWriter, reader DReader, keysDeduplication Dedup, back ...Storage) *redundant

Redundant storage with custom strategy for writing and reading backed by several storage

func RedundantAll

func RedundantAll(keysDeduplication Dedup, back ...Storage) *redundant

Redundant storage that writes values to all back storages and read from first successful.

Types

type Accessor

type Accessor interface {
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
	// Put single item to storage. If already exists - override
	Put(key []byte, data []byte) error
	// Delete key and value
	Del(key []byte) error
}

Access only interface for storage

type BatchedStorage

type BatchedStorage interface {
	Storage
	BatchWriter() Writer
}

Atomic (batch) writer. Batch storage should be used only in one thread

type Clearable

type Clearable interface {
	// Clear all data in storage
	Clear() error
}

Clear storage

type DReader

type DReader func(key []byte, storages []Storage) ([]byte, error)

Distributed reader strategy

func First

func First() DReader

First non-empty value for key will be used as result

type DWriter

type DWriter func(key, data []byte, storages []Storage) error

Distributed writer strategy

func Any

func Any() DWriter

Shorthand for AtLeast(1) - requires at least one successful write operation

func AtLeast

func AtLeast(minWrite int) DWriter

At least minWrite amount of written operations should be complete for success result

type Dedup

type Dedup interface {
	// Is key already save?
	IsDuplicated(key []byte) (bool, error)
	// Save key for future checks
	Save(key []byte) error
}

Deduplicate primitive: check if key is already saved and save key

type Iterator

type Iterator interface {
	// Is queue has next value
	Next() bool
	// Current id
	ID() int64
	// Current value
	Value() []byte
}

Queue iterator

type KV

type KV interface {
	Writer
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
	// Delete key and value
	Del(key []byte) error
}

Thread-safe storage for key-value

type LegacyQueue

type LegacyQueue interface {
	// put data to queue using new sequence id
	Put(data []byte) (id int64, err error)
	// peek latest data
	Peek() (id int64, data []byte, err error)
	// clean data in range: [first;end)
	Clean(end int64) error
	// size of queue (last-first)
	Size() int64
	// first (oldest) sequence id
	First() int64
	// last (latest) sequence id
	Last() int64
	// iterate over items from first (if from is 0) to last (next should be called first) or till first error.
	// Iterator keeps min and max sequence number so cleaning items during iteration may cause iteration stop
	Iterate(from int64) Iterator
}

Wrapper around storage that makes sequential data inserting and peeking Without external access to the data Queue guarantees that sequences are without space and strictly increasing. If queue has no data sequence is flushed (starts from 0) after restart

type LimitedQueue

type LimitedQueue interface {
	LegacyQueue
	// available space (limit - size)
	Available() int64
	// limit
	Limit() int64
}

Queue with limited size

type NamespacedStorage

type NamespacedStorage interface {
	Storage
	// Get or create nested storage. Optionally can be also NamespacedStorage but it is implementation defined
	Namespace(name []byte) (Storage, error)
	// Iterate over all namespaces in storage (not including nested)
	Namespaces(handler func(name []byte) error) error
	// Delete nested namespace by name. If namespace still in usage - result undefined
	DelNamespace(name []byte) error
}

Nested storage with namespace support (implementation defined). Namespaces and regular values may live in a same key-space.

type Queue

type Queue interface {
	// Put data to the queue
	Put(data []byte) error
	// Get oldest record but not remove it. Returns os.ErrNotExists if queue is empty
	Peek() ([]byte, error)
	// Discard oldest record. Returns os.ErrNotExists if queue is empty
	Discard() error
	// Get oldest record and remove it.  Returns os.ErrNotExists if queue is empty. Basically - it's atomic version of Peek & Discard
	Get() ([]byte, error)
}

Thread safe queue

type Reader

type Reader interface {
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
}

Key-value reader

type ShardPool

type ShardPool interface {
	// Get storage based on key
	Get(key []byte) (Storage, error)
	// Iterate over shards
	Iterate(handler func(storage Storage) error) error
	io.Closer
}

Sharding pool

type Storage

type Storage interface {
	KV
	// Iterate over all keys. Modification during iteration may cause undefined behaviour (mostly - dead-lock)
	Keys(handler func(key []byte) error) error
}

Extension for KV storage with iterator over keys

func Compressed

func Compressed(storage Storage) Storage

Compressed storage where values are compressed by gzip

func Sharded

func Sharded(pool ShardPool) Storage

Sharded storage with defined pool (strategy)

type Writer

type Writer interface {
	// Put single item to storage. If already exists - override
	Put(key []byte, data []byte) error
	// Close storage if needs
	io.Closer
}

Key-value writer