datastore

package module
v0.0.0-...-e63957b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 3, 2016 License: MIT Imports: 7 Imported by: 0

README

datastore interface

datastore is a generic layer of abstraction for data store and database access. It is a simple API with the aim to enable application development in a datastore-agnostic way, allowing datastores to be swapped seamlessly without changing application code. Thus, one can leverage different datastores with different strengths without committing the application to one datastore throughout its lifetime.

In addition, grouped datastores significantly simplify interesting data access patterns (such as caching and sharding).

Based on datastore.py.

Documentation

https://godoc.org/github.com/jbenet/go-datastore

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrBatchUnsupported = errors.New("this datastore does not support batching")
View Source 
var ErrInvalidType = errors.New("datastore: invalid type error")

ErrInvalidType is returned by Put when a given value is incopatible with the type the datastore supports. This means a conversion (or serialization) is needed beforehand.

View Source 
var ErrNotFound = errors.New("datastore: key not found")

ErrNotFound is returned by Get, Has, and Delete when a datastore does not map the given key to a value.

Functions

func GetBackedHas 

func GetBackedHas(ds Datastore, key Key) (bool, error)

GetBackedHas provides a default Datastore.Has implementation. It exists so Datastore.Has implementations can use it, like so: 

func (*d SomeDatastore) Has(key Key) (exists bool, err error) {
  return GetBackedHas(d, key)
}

func NamespaceType 

func NamespaceType(namespace string) string

NamespaceType is the first component of a namespace. `foo` in `foo:bar`

func NamespaceValue 

func NamespaceValue(namespace string) string

NamespaceValue returns the last component of a namespace. `baz` in `f:b:baz`

Types

type Batch 

type Batch interface {
	Put(key Key, val interface{}) error

	Delete(key Key) error

	Commit() error
}

func NewBasicBatch 

func NewBasicBatch(ds Datastore) Batch

type Batching 

type Batching interface {
	Datastore

	Batch() (Batch, error)
}

type Datastore 

type Datastore interface {
	// Put stores the object `value` named by `key`.
	//
	// The generalized Datastore interface does not impose a value type,
	// allowing various datastore middleware implementations (which do not
	// handle the values directly) to be composed together.
	//
	// Ultimately, the lowest-level datastore will need to do some value checking
	// or risk getting incorrect values. It may also be useful to expose a more
	// type-safe interface to your application, and do the checking up-front.
	Put(key Key, value interface{}) error

	// Get retrieves the object `value` named by `key`.
	// Get will return ErrNotFound if the key is not mapped to a value.
	Get(key Key) (value interface{}, err error)

	// Has returns whether the `key` is mapped to a `value`.
	// In some contexts, it may be much cheaper only to check for existence of
	// a value, rather than retrieving the value itself. (e.g. HTTP HEAD).
	// The default implementation is found in `GetBackedHas`.
	Has(key Key) (exists bool, err error)

	// Delete removes the value for given `key`.
	Delete(key Key) error

	// Query searches the datastore and returns a query result. This function
	// may return before the query actually runs. To wait for the query:
	//
	//   result, _ := ds.Query(q)
	//
	//   // use the channel interface; result may come in at different times
	//   for entry := range result.Entries() { ... }
	//
	//	 // or wait for the query to be completely done
	//   result.Wait()
	//   result.AllEntries()
	//
	Query(q query.Query) (query.Results, error)
}

Datastore represents storage for any key-value pair.

Datastores are general enough to be backed by all kinds of different storage: in-memory caches, databases, a remote datastore, flat files on disk, etc.

The general idea is to wrap a more complicated storage facility in a simple, uniform interface, keeping the freedom of using the right tools for the job. In particular, a Datastore can aggregate other datastores in interesting ways, like sharded (to distribute load) or tiered access (caches before databases).

While Datastores should be written general enough to accept all sorts of values, some implementations will undoubtedly have to be specific (e.g. SQL databases where fields should be decomposed into columns), particularly to support queries efficiently. Moreover, certain datastores may enforce certain types of values (e.g. requiring an io.Reader, a specific struct, etc) or serialization formats (JSON, Protobufs, etc).

IMPORTANT: No Datastore should ever Panic! This is a cross-module interface, and thus it should behave predictably and handle exceptional conditions with proper error reporting. Thus, all Datastore calls may return errors, which should be checked by callers.

type Key 

type Key struct {
	// contains filtered or unexported fields
}

A Key represents the unique identifier of an object. Our Key scheme is inspired by file systems and Google App Engine key model.

Keys are meant to be unique across a system. Keys are hierarchical, incorporating more and more specific namespaces. Thus keys can be deemed 'children' or 'ancestors' of other keys:: 

Key("/Comedy")
Key("/Comedy/MontyPython")

Also, every namespace can be parametrized to embed relevant object information. For example, the Key `name` (most specific namespace) could include the object type:: 

Key("/Comedy/MontyPython/Actor:JohnCleese")
Key("/Comedy/MontyPython/Sketch:CheeseShop")
Key("/Comedy/MontyPython/Sketch:CheeseShop/Character:Mousebender")

func EntryKeys 

func EntryKeys(e []dsq.Entry) []Key

EntryKeys

func KeyWithNamespaces 

func KeyWithNamespaces(ns []string) Key

KeyWithNamespaces constructs a key out of a namespace slice.

func NewKey 

func NewKey(s string) Key

NewKey constructs a key from string. it will clean the value.

func RandomKey 

func RandomKey() Key

RandomKey returns a randomly (uuid) generated key. 

RandomKey()
NewKey("/f98719ea086343f7b71f32ea9d9d521d")

func (Key) BaseNamespace 

func (k Key) BaseNamespace() string

BaseNamespace returns the "base" namespace of this key (path.Base(filename)) 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").BaseNamespace()
"Actor:JohnCleese"

func (Key) Bytes 

func (k Key) Bytes() []byte

Bytes returns the string value of Key as a []byte

func (Key) Child 

func (k Key) Child(k2 Key) Key

Child returns the `child` Key of this Key. 

NewKey("/Comedy/MontyPython").Child("Actor:JohnCleese")
NewKey("/Comedy/MontyPython/Actor:JohnCleese")

func (Key) ChildString 

func (k Key) ChildString(s string) Key

ChildString returns the `child` Key of this Key -- string helper. 

NewKey("/Comedy/MontyPython").Child("Actor:JohnCleese")
NewKey("/Comedy/MontyPython/Actor:JohnCleese")

func (*Key) Clean 

func (k *Key) Clean()

Clean up a Key, using path.Clean.

func (Key) Equal 

func (k Key) Equal(k2 Key) bool

Equal checks equality of two keys

func (Key) Instance 

func (k Key) Instance(s string) Key

Instance returns an "instance" of this type key (appends value to namespace). 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").List()
"JohnCleese"

func (Key) IsAncestorOf 

func (k Key) IsAncestorOf(other Key) bool

IsAncestorOf returns whether this key is a prefix of `other` 

NewKey("/Comedy").IsAncestorOf("/Comedy/MontyPython")
true

func (Key) IsDescendantOf 

func (k Key) IsDescendantOf(other Key) bool

IsDescendantOf returns whether this key contains another as a prefix. 

NewKey("/Comedy/MontyPython").IsDescendantOf("/Comedy")
true

func (Key) IsTopLevel 

func (k Key) IsTopLevel() bool

IsTopLevel returns whether this key has only one namespace.

func (Key) Less 

func (k Key) Less(k2 Key) bool

Less checks whether this key is sorted lower than another.

func (Key) List 

func (k Key) List() []string

List returns the `list` representation of this Key. 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").List()
["Comedy", "MontyPythong", "Actor:JohnCleese"]

func (Key) Name 

func (k Key) Name() string

Name returns the "name" of this key (field of last namespace). 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").List()
"Actor"

func (Key) Namespaces 

func (k Key) Namespaces() []string

Namespaces returns the `namespaces` making up this Key. 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").List()
["Comedy", "MontyPythong", "Actor:JohnCleese"]

func (Key) Parent 

func (k Key) Parent() Key

Parent returns the `parent` Key of this Key. 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Parent()
NewKey("/Comedy/MontyPython")

func (Key) Path 

func (k Key) Path() Key

Path returns the "path" of this key (parent + type). 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Path()
NewKey("/Comedy/MontyPython/Actor")

func (Key) Reverse 

func (k Key) Reverse() Key

Reverse returns the reverse of this Key. 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").Reverse()
NewKey("/Actor:JohnCleese/MontyPython/Comedy")

func (Key) String 

func (k Key) String() string

Strings is the string value of Key

func (Key) Type 

func (k Key) Type() string

Type returns the "type" of this key (value of last namespace). 

NewKey("/Comedy/MontyPython/Actor:JohnCleese").List()
"Actor"

type KeySlice 

type KeySlice []Key

KeySlice attaches the methods of sort.Interface to []Key, sorting in increasing order.

func (KeySlice) Len 

func (p KeySlice) Len() int

func (KeySlice) Less 

func (p KeySlice) Less(i, j int) bool

func (KeySlice) Swap 

func (p KeySlice) Swap(i, j int)

type LogDatastore 

type LogDatastore struct {
	Name string
	// contains filtered or unexported fields
}

LogDatastore logs all accesses through the datastore.

func NewLogDatastore 

func NewLogDatastore(ds Datastore, name string) *LogDatastore

NewLogDatastore constructs a log datastore.

func (*LogDatastore) Batch 

func (d *LogDatastore) Batch() (Batch, error)

func (*LogDatastore) Children 

func (d *LogDatastore) Children() []Datastore

Children implements Shim

func (*LogDatastore) Close 

func (d *LogDatastore) Close() error

func (*LogDatastore) Delete 

func (d *LogDatastore) Delete(key Key) (err error)

Delete implements Datastore.Delete

func (*LogDatastore) Get 

func (d *LogDatastore) Get(key Key) (value interface{}, err error)

Get implements Datastore.Get

func (*LogDatastore) Has 

func (d *LogDatastore) Has(key Key) (exists bool, err error)

Has implements Datastore.Has

func (*LogDatastore) Put 

func (d *LogDatastore) Put(key Key, value interface{}) (err error)

Put implements Datastore.Put

func (*LogDatastore) Query 

func (d *LogDatastore) Query(q dsq.Query) (dsq.Results, error)

Query implements Datastore.Query

type MapDatastore 

type MapDatastore struct {
	// contains filtered or unexported fields
}

MapDatastore uses a standard Go map for internal storage.

func NewMapDatastore 

func NewMapDatastore() (d *MapDatastore)

NewMapDatastore constructs a MapDatastore

func (*MapDatastore) Batch 

func (d *MapDatastore) Batch() (Batch, error)

func (*MapDatastore) Close 

func (d *MapDatastore) Close() error

func (*MapDatastore) Delete 

func (d *MapDatastore) Delete(key Key) (err error)

Delete implements Datastore.Delete

func (*MapDatastore) Get 

func (d *MapDatastore) Get(key Key) (value interface{}, err error)

Get implements Datastore.Get

func (*MapDatastore) Has 

func (d *MapDatastore) Has(key Key) (exists bool, err error)

Has implements Datastore.Has

func (*MapDatastore) Put 

func (d *MapDatastore) Put(key Key, value interface{}) (err error)

Put implements Datastore.Put

func (*MapDatastore) Query 

func (d *MapDatastore) Query(q dsq.Query) (dsq.Results, error)

Query implements Datastore.Query

type NullDatastore 

type NullDatastore struct {
}

NullDatastore stores nothing, but conforms to the API. Useful to test with.

func NewNullDatastore 

func NewNullDatastore() *NullDatastore

NewNullDatastore constructs a null datastoe

func (*NullDatastore) Batch 

func (d *NullDatastore) Batch() (Batch, error)

func (*NullDatastore) Close 

func (d *NullDatastore) Close() error

func (*NullDatastore) Delete 

func (d *NullDatastore) Delete(key Key) (err error)

Delete implements Datastore.Delete

func (*NullDatastore) Get 

func (d *NullDatastore) Get(key Key) (value interface{}, err error)

Get implements Datastore.Get

func (*NullDatastore) Has 

func (d *NullDatastore) Has(key Key) (exists bool, err error)

Has implements Datastore.Has

func (*NullDatastore) Put 

func (d *NullDatastore) Put(key Key, value interface{}) (err error)

Put implements Datastore.Put

func (*NullDatastore) Query 

func (d *NullDatastore) Query(q dsq.Query) (dsq.Results, error)

Query implements Datastore.Query

type Shim 

type Shim interface {
	Datastore

	Children() []Datastore
}

Shim is a datastore which has a child.

type ThreadSafeDatastore 

type ThreadSafeDatastore interface {
	Datastore
	IsThreadSafe()
}

ThreadSafeDatastore is an interface that all threadsafe datastore should implement to leverage type safety checks.

Source Files

View all Source files

Directories

Path Synopsis
Godeps
_workspace/src/github.com/Sirupsen/logrus
Package logrus is a structured logger for Go, completely API compatible with the standard library logger.
 Package logrus is a structured logger for Go, completely API compatible with the standard library logger.
_workspace/src/github.com/Sirupsen/logrus/examples/basic
_workspace/src/github.com/Sirupsen/logrus/examples/hook
_workspace/src/github.com/Sirupsen/logrus/formatters/logstash
_workspace/src/github.com/Sirupsen/logrus/hooks/airbrake
_workspace/src/github.com/Sirupsen/logrus/hooks/bugsnag
_workspace/src/github.com/Sirupsen/logrus/hooks/papertrail
_workspace/src/github.com/Sirupsen/logrus/hooks/sentry
_workspace/src/github.com/Sirupsen/logrus/hooks/syslog
_workspace/src/github.com/codahale/blake2
Package blake2 provides a Go wrapper around an optimized, public domain implementation of BLAKE2.
 Package blake2 provides a Go wrapper around an optimized, public domain implementation of BLAKE2.
_workspace/src/github.com/codahale/hdrhistogram
Package hdrhistogram provides an implementation of Gil Tene's HDR Histogram data structure.
 Package hdrhistogram provides an implementation of Gil Tene's HDR Histogram data structure.
_workspace/src/github.com/codahale/metrics
Package metrics provides minimalist instrumentation for your applications in the form of counters and gauges.
 Package metrics provides minimalist instrumentation for your applications in the form of counters and gauges.
_workspace/src/github.com/codahale/metrics/runtime
Package runtime registers gauges and counters for various operationally important aspects of the Go runtime.
 Package runtime registers gauges and counters for various operationally important aspects of the Go runtime.
_workspace/src/github.com/dustin/randbo
_workspace/src/github.com/fzzy/radix/redis
A simple client for connecting and interacting with redis.
 A simple client for connecting and interacting with redis.
_workspace/src/github.com/fzzy/radix/redis/resp
This package provides an easy to use interface for creating and parsing messages encoded in the REdis Serialization Protocol (RESP).
 This package provides an easy to use interface for creating and parsing messages encoded in the REdis Serialization Protocol (RESP).
_workspace/src/github.com/hashicorp/golang-lru
This package provides a simple LRU cache.
 This package provides a simple LRU cache.
_workspace/src/github.com/ipfs/go-log
_workspace/src/github.com/jbenet/go-os-rename
_workspace/src/github.com/jbenet/goprocess
Package goprocess introduces a Process abstraction that allows simple organization, and orchestration of work.
 Package goprocess introduces a Process abstraction that allows simple organization, and orchestration of work.
_workspace/src/github.com/jbenet/goprocess/context
_workspace/src/github.com/jbenet/goprocess/periodic
Package periodic is part of github.com/jbenet/goprocess.
 Package periodic is part of github.com/jbenet/goprocess.
_workspace/src/github.com/jbenet/goprocess/ratelimit
Package ratelimit is part of github.com/jbenet/goprocess.
 Package ratelimit is part of github.com/jbenet/goprocess.
_workspace/src/github.com/mattbaird/elastigo/api
_workspace/src/github.com/mattbaird/elastigo/core
_workspace/src/github.com/satori/go.uuid
Package uuid provides implementation of Universally Unique Identifier (UUID).
 Package uuid provides implementation of Universally Unique Identifier (UUID).
_workspace/src/github.com/syndtr/goleveldb/leveldb
Package leveldb provides implementation of LevelDB key/value database.
 Package leveldb provides implementation of LevelDB key/value database.
_workspace/src/github.com/syndtr/goleveldb/leveldb/cache
Package cache provides interface and implementation of a cache algorithms.
 Package cache provides interface and implementation of a cache algorithms.
_workspace/src/github.com/syndtr/goleveldb/leveldb/comparer
Package comparer provides interface and implementation for ordering sets of data.
 Package comparer provides interface and implementation for ordering sets of data.
_workspace/src/github.com/syndtr/goleveldb/leveldb/errors
Package errors provides common error types used throughout leveldb.
 Package errors provides common error types used throughout leveldb.
_workspace/src/github.com/syndtr/goleveldb/leveldb/filter
Package filter provides interface and implementation of probabilistic data structure.
 Package filter provides interface and implementation of probabilistic data structure.
_workspace/src/github.com/syndtr/goleveldb/leveldb/iterator
Package iterator provides interface and implementation to traverse over contents of a database.
 Package iterator provides interface and implementation to traverse over contents of a database.
_workspace/src/github.com/syndtr/goleveldb/leveldb/journal
Package journal reads and writes sequences of journals.
 Package journal reads and writes sequences of journals.
_workspace/src/github.com/syndtr/goleveldb/leveldb/memdb
Package memdb provides in-memory key/value database implementation.
 Package memdb provides in-memory key/value database implementation.
_workspace/src/github.com/syndtr/goleveldb/leveldb/opt
Package opt provides sets of options used by LevelDB.
 Package opt provides sets of options used by LevelDB.
_workspace/src/github.com/syndtr/goleveldb/leveldb/storage
Package storage provides storage abstraction for LevelDB.
 Package storage provides storage abstraction for LevelDB.
_workspace/src/github.com/syndtr/goleveldb/leveldb/table
Package table allows read and write sorted key/value.
 Package table allows read and write sorted key/value.
_workspace/src/github.com/syndtr/goleveldb/leveldb/testutil
_workspace/src/github.com/syndtr/goleveldb/leveldb/util
Package util provides utilities used throughout leveldb.
 Package util provides utilities used throughout leveldb.
_workspace/src/github.com/syndtr/gosnappy/snappy
Package snappy implements the snappy block-based compression format.
 Package snappy implements the snappy block-based compression format.
_workspace/src/golang.org/x/net/context
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
 Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
_workspace/src/gopkg.in/check.v1
Package check is a rich testing extension for Go's testing package.
 Package check is a rich testing extension for Go's testing package.
_workspace/src/launchpad.net/gocheck
Package flatfs is a Datastore implementation that stores all objects in a two-level directory structure in the local file system, regardless of the hierarchy of the keys.
 Package flatfs is a Datastore implementation that stores all objects in a two-level directory structure in the local file system, regardless of the hierarchy of the keys.
Package fs is a simple Datastore implementation that stores keys are directories and files, mirroring the key.
 Package fs is a simple Datastore implementation that stores keys are directories and files, mirroring the key.
Package keytransform introduces a Datastore Shim that transforms keys before passing them to its child.
 Package keytransform introduces a Datastore Shim that transforms keys before passing them to its child.
Package measure provides a Datastore wrapper that records metrics using github.com/codahale/metrics.
 Package measure provides a Datastore wrapper that records metrics using github.com/codahale/metrics.
Package mount provides a Datastore that has other Datastores mounted at various key prefixes.
 Package mount provides a Datastore that has other Datastores mounted at various key prefixes.
Package namespace introduces a namespace Datastore Shim, which basically mounts the entire child datastore under a prefix.
 Package namespace introduces a namespace Datastore Shim, which basically mounts the entire child datastore under a prefix.
Package mount provides a Datastore that has other Datastores mounted at various key prefixes and is threadsafe
 Package mount provides a Datastore that has other Datastores mounted at various key prefixes and is threadsafe

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.