casper

README

Casper is a library for arbitrary key-value or content addressed data storage implemented with modular, composable layers. There are back-end storage drivers for different interfaces, an optional content-addressed front-end, and tranformation layers implementing features like transparent compression and content defined chunking.

This library is under very early stages of development and is not intended to provide a stable API yet.

Key-Value Store Interface

The kvs.Store interface defines a minimal interface for arbitrary key-value storage drivers:

type Store interface {
	Exists(key string) (bool, error)
	Read(key string) ([]byte, error)
	ReadStream(key string) (io.ReadCloser, error)
	Write(key string, data []byte) error
	WriteStream(key string, reader io.Reader) error
	Remove(key string) error
}

Existing keys may not be written to, but may be deleted and recreated with different contents. Read-only drivers may be defined and accessed with the kvs.StoreReader interface.

Currently implemented storage drivers include:

• fs native filesystem disk-based storage.
• httpremote driver for read-only HTTP remotes.
• sshremote driver using pkg/sftp for read-write remote filesystem access.
• memory very simple map based storage for testing.

Content Addressed Storage Front-End

The cas.Store interface is nearly identical to the to the kvs.Store but with the requirement that the key be the output of a hash function applied to the input data. Since the key for any given record is static, an optional file extension is accepted which will be appended to the key name to explicitly define a file type.

type Store interface {
	Exists(key, extension string) (bool, error)
	Read(key, extension string) ([]byte, error)
	ReadStream(key, extension string) (io.ReadCloser, error)
	Write(key, extension string, data []byte) error
	WriteStream(key, extension string, reader io.Reader) error
}

Chunking Layers

There is also a chunking layer that will split files into chunks. A simple interface is defined that can be created from an io.Reader to define a chunking algorithm.

An example Chunker implementation is provided using the restic chunker package which implements content-defined-chunking using a rolling hash. Using this transformation, small changes to a file will generally only affect the chunk(s) near the change, while the remaining chunks in the file will stay the same, reducing storage and bandwidth requirements.

See this blog post for an introduction to Content Defined Chunking.

Transparent Compression Layers

Transparent compression (or any other streaming, reversible encoding) may be added using the Transcoder interface:

type Transcoder interface {
	NewWriter(io.Writer) (io.WriteCloser, error)
	NewReader(io.ReadCloser) (io.ReadCloser, error)
}

The following example implementations are included in kvs/compression:

• gzip from standard library.
• zstd from pure go klauspost/compress