store

Documentation

Overview

Package store provides a simple, goroutine safe key-value interface. Instead of values being an opaque array of bytes, though, they are a stream. This approach allows large items to be stored easily.

Probably the most important implementation is the FileSystem. The other are stores are useful for testing or other specialized situations.

Index

• Variables
• func NewReader(r io.ReaderAt) io.Reader
• type BlackPearl
  • func NewBlackPearl(bucket, prefix string, client *ds3.Client) *BlackPearl
  • func (bp *BlackPearl) Create(key string) (io.WriteCloser, error)
  • func (bp *BlackPearl) Delete(key string) error
  • func (bp *BlackPearl) List() <-chan string
  • func (bp *BlackPearl) ListPrefix(prefix string) ([]string, error)
  • func (bp *BlackPearl) Open(key string) (ReadAtCloser, int64, error)
  • func (bp *BlackPearl) Stage(keys []string)
• type COW
  • func NewCOW(local Store, host, token string) *COW
  • func (c *COW) Create(key string) (io.WriteCloser, error)
  • func (c *COW) Delete(key string) error
  • func (c *COW) List() <-chan string
  • func (c *COW) ListPrefix(prefix string) ([]string, error)
  • func (c *COW) Open(key string) (ReadAtCloser, int64, error)
• type FileSystem
  • func NewFileSystem(root string) *FileSystem
  • func (s *FileSystem) Create(key string) (io.WriteCloser, error)
  • func (s *FileSystem) Delete(key string) error
  • func (s *FileSystem) List() <-chan string
  • func (s *FileSystem) ListPrefix(prefix string) ([]string, error)
  • func (s *FileSystem) Open(key string) (ReadAtCloser, int64, error)
• type Memory
  • func NewMemory() *Memory
  • func (ms *Memory) Create(key string) (io.WriteCloser, error)
  • func (ms *Memory) Delete(key string) error
  • func (ms *Memory) Dump(w io.Writer)
  • func (ms *Memory) List() <-chan string
  • func (ms *Memory) ListPrefix(prefix string) ([]string, error)
  • func (ms *Memory) Open(key string) (ReadAtCloser, int64, error)
• type ROStore
• type ReadAtCloser
• type S3
  • func NewS3(bucket, prefix string, awsSession *session.Session) *S3
  • func (s *S3) Create(key string) (io.WriteCloser, error)
  • func (s *S3) Delete(key string) error
  • func (s *S3) List() <-chan string
  • func (s *S3) ListPrefix(prefix string) ([]string, error)
  • func (s *S3) Open(key string) (ReadAtCloser, int64, error)
• type Stager
• type Store
  • func NewWithPrefix(s Store, prefix string) Store

Variables

var (

	// ErrKeyExists indicates an attempt to create a key which already exists
	ErrKeyExists = errors.New("Key already exists")

	// ErrKeyContainsSlash means the key provided contains a forward slash '/'
	ErrKeyContainsSlash = errors.New("Key contains forward slash")

	// ErrKeyContainsNonUnicode means the key provided contains a Non Unicode Rune
	ErrKeyContainsNonUnicode = errors.New("Key contains Non-Unicode character")

	// ErrKeyContainsWhiteSpace  means the key provided contains WhiteSpace
	ErrKeyContainsWhiteSpace = errors.New("Key contains White Space")

	// ErrKeyContainsControlChar  means the key provided contains Control Characters
	ErrKeyContainsControlChar = errors.New("Key contains Control  Characters")
)

var (
	ErrNoETag   = errors.New("No ETag was returned from AWS")
	ErrNotExist = errors.New("Key does not exist")
)

Functions

func NewReader

func NewReader(r io.ReaderAt) io.Reader

NewReader converts a ReaderAt into a io.Reader. It is here as a utility to help work with the ReadAtCloser returned by Open.

Types

type BlackPearl

type BlackPearl struct {
	Bucket  string
	Prefix  string
	TempDir string // where to make temp files. "" uses default place
	// contains filtered or unexported fields
}

A BlackPearl store represents a store that is kept on a SpectraLogic's BlackPearl appliance. Do not change Bucket or Prefix concurrently with calls using the structure.

func NewBlackPearl

func NewBlackPearl(bucket, prefix string, client *ds3.Client) *BlackPearl

NewBlackPearl creates a new BlackPearl store. It will use the given bucket and will prepend prefix to all keys. This is to allow for a bucket to be used for more than one store. For example if prefix were "cache/" then an Open("hello") would look for the key "cache/hello" in the bucket. The authorization method and credentials in the session are used for all accesses.

func (*BlackPearl) Create

func (bp *BlackPearl) Create(key string) (io.WriteCloser, error)

Create will return a WriteCloser to upload content to the given key. Data is batched into a temporary file and then uploaded when Close() is called. Might need to make directoy of the temporary file be configurable.

func (*BlackPearl) Delete

func (bp *BlackPearl) Delete(key string) error

Delete will remove the given key from the store. The store's Prefix is prepended first. It is not an error to delete something that doesn't exist.

func (*BlackPearl) List

func (bp *BlackPearl) List() <-chan string

List returns a list of all the keys in this store. It will only return ones that satisfy the store's Prefix, so it is safe to use this on a bucket containing other items.

func (*BlackPearl) ListPrefix

func (bp *BlackPearl) ListPrefix(prefix string) ([]string, error)

ListPrefix returns the keys in this store that have the given prefix. The argument prefix is added to the store's Prefix.

func (*BlackPearl) Open

func (bp *BlackPearl) Open(key string) (ReadAtCloser, int64, error)

Open will return a ReadAtCloser to get the content for the given key.

The current implementation will download the entire contents for every call to Open(). There is a lot of room for optimization and caching.

func (*BlackPearl) Stage

func (bp *BlackPearl) Stage(keys []string)

Stage will take a list of keys and ask the BlackPearl to stage all of them in its cache. It is for performance only and doesn't need to be called.

type COW

type COW struct {
	// contains filtered or unexported fields
}

COW implements a Copy-on-Write store multiplexing between an external bendo host and a local store. The local store is used for writes and is the first checked for reads. Anything not in the local store will then be looked up in the external one. Hence, the local store appears to have everything in the external one, but all changes are local and nothing is ever written to the external store.

Note: While simple, this is not the most efficient way of implementing a COW interface. To read a single file, this store will first copy an entire bundle from the external host, and then read out only the one file which was requested.

I consider this approach a proof of concept. The "for real" approach will be to alter Item to make the appropriate calls into the remote bendo server.

func NewCOW

func NewCOW(local Store, host, token string) *COW

NewCOW creates a COW store using the given local store and the given external host. Host should be in form "http://hostname:port". The optional token is the user token to use in requests to the server.

func (*COW) Create

func (c *COW) Create(key string) (io.WriteCloser, error)

Create will make a new item in the local store. It is acceptable to make an item in the local store with the same name as an item in the remote store. The local item will shadow the remote one.

func (*COW) Delete

func (c *COW) Delete(key string) error

Delete `key`. Items will only be deleted from the local store. Trying to delete a remote item will result in a nop (but not an error). Note: If there were a local item shadowing a remote item, doing a delete will delete the local one, but the remote one will still exist. So deleting an item may not remote it from the store. This semantics may cause problems with users expecting a standard store.

func (*COW) List

func (c *COW) List() <-chan string

List returns a channel enumerating everything in this store. It will combine the items in both the local store and the remote store.

func (*COW) ListPrefix

func (c *COW) ListPrefix(prefix string) ([]string, error)

ListPrefix returns all items with a specified prefix. It will combine the items found from both stores.

func (*COW) Open

func (c *COW) Open(key string) (ReadAtCloser, int64, error)

Open will return an item for reading. If the item is only in the remote store, it will first be copied into the local store.

type FileSystem

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem implements the simple file system based store. It tries to only open files when necessary, so it could be backed by a tape system, for example. The keys are used as file names. This means keys should not contain a forward slash character '/'. Also, if you want the files to have a specific file extension, you need to add it to your key.

func NewFileSystem

func NewFileSystem(root string) *FileSystem

NewFileSystem creates a new FileSystem store based at the given root path.

func (*FileSystem) Create

func (s *FileSystem) Create(key string) (io.WriteCloser, error)

Create creates a new item with the given key, and a writer to allow for saving data into the new item.

func (*FileSystem) Delete

func (s *FileSystem) Delete(key string) error

Delete the given key from the store. It is not an error if the key doesn't exist.

func (*FileSystem) List

func (s *FileSystem) List() <-chan string

List returns a channel listing all the keys in this store.

func (*FileSystem) ListPrefix

func (s *FileSystem) ListPrefix(prefix string) ([]string, error)

ListPrefix returns a list of all the keys beginning with the given prefix.

func (*FileSystem) Open

func (s *FileSystem) Open(key string) (ReadAtCloser, int64, error)

Open returns a reader for the given object along with its size.

type Memory

type Memory struct {
	// contains filtered or unexported fields
}

Memory implements a simple in-memory version of a store. It is intended mainly for testing.

func NewMemory

func NewMemory() *Memory

NewMemory returns a new, empty memory store.

func (*Memory) Create

func (ms *Memory) Create(key string) (io.WriteCloser, error)

Create makes a new entry in the store, and returns a writer to save data into it.

func (*Memory) Delete

func (ms *Memory) Delete(key string) error

Delete the given key from the store. It is not an error if the item does not exist in the store.

func (*Memory) Dump

func (ms *Memory) Dump(w io.Writer)

Dump writes a listing of the contents of the store to the given writer. This is intended for testing and debugging.

func (*Memory) List

func (ms *Memory) List() <-chan string

List returns a channel giving the id for every item in the store.

The goroutine started to generate the list keeps a read lock on the underlying store for its duration. This may cause deadlocks.

func (*Memory) ListPrefix

func (ms *Memory) ListPrefix(prefix string) ([]string, error)

ListPrefix returns all the key entries which begin with the given prefix.

func (*Memory) Open

func (ms *Memory) Open(key string) (ReadAtCloser, int64, error)

Open returns a ReadAtCloser and the size of the given blob.

type ROStore

type ROStore interface {
	List() <-chan string
	ListPrefix(prefix string) ([]string, error)
	Open(key string) (ReadAtCloser, int64, error)
}

ROStore is the read-only pieces of a Store. It allows one to list contents, and to retrieve data.

type ReadAtCloser

type ReadAtCloser interface {
	io.ReaderAt
	io.Closer
}

ReadAtCloser combines the io.ReaderAt and io.Closer interfaces.

type S3

type S3 struct {
	Bucket string
	Prefix string
	// contains filtered or unexported fields
}

A S3 store represents a store that is kept on AWS S3 storage. Do not change Bucket or Prefix concurrently with calls using the structure.

func NewS3

func NewS3(bucket, prefix string, awsSession *session.Session) *S3

NewS3 creates a new S3 store. It will use the given bucket and will prepend prefix to all keys. This is to allow for a bucket to be used for more than one store. For example if prefix were "cache/" then an Open("hello") would look for the key "cache/hello" in the bucket. The authorization method and credentials in the session are used for all accesses.

func (*S3) Create

func (s *S3) Create(key string) (io.WriteCloser, error)

Create will return a WriteCloser to upload content to the given key. Data is batched and uploaded to S3 using the Multipart interface. The part sizes increase, so objects up to the 5 TB limit S3 imposes is theoretically possible.

func (*S3) Delete

func (s *S3) Delete(key string) error

Delete will remove the given key from the store. The store's Prefix is prepended first. It is not an error to delete something that doesn't exist.

func (*S3) List

func (s *S3) List() <-chan string

List returns a list of all the keys in this store. It will only return ones that satisfy the store's Prefix, so it is safe to use this on a bucket containing other items.

func (*S3) ListPrefix

func (s *S3) ListPrefix(prefix string) ([]string, error)

ListPrefix returns the keys in this store that have the given prefix. The argument prefix is added to the store's Prefix.

func (*S3) Open

func (s *S3) Open(key string) (ReadAtCloser, int64, error)

Open will return a ReadAtCloser to get the content for the given key. Data is paged in from S3 as needed, and up to 50 MB or so is cached at a time.

type Stager

type Stager interface {
	Stage(keys []string)
}

Stager is something where it can get a performace gain from prefitching files. Stage() should not affect the semantics of the store in any other way.

type Store

type Store interface {
	ROStore
	Create(key string) (io.WriteCloser, error)
	Delete(key string) error
}

Store defines the basic stream based key-value store. Items are immutable once stored, but they may be deleted and then replaced with a new value.

Since the FileSystem store uses the key as file names, keys should not contain forbidden filesystem characters, such as '/'.

Open() returns a ReadAtCloser instead of a ReadCloser to make it easier to wrap it by a ZipWriter.

TODO: is a Close() method needed?

func NewWithPrefix

func NewWithPrefix(s Store, prefix string) Store

NewWithPrefix wraps a store by one which will prefix all its keys with the prefix. This provides a basic way to namespace the keys, and to share the same underlying store among a group of distinct stores.