nds

README

nds (v2 - EXPERIMENTAL)

Package github.com/qedus/nds is a Google Cloud Datastore API for Go that uses a cache backend to cache all datastore requests. Memcache is only supported on Google AppEngine Standard, but the package can be used for any other implemented cache backend on any platform (local, Google Compute, AWS, etc.). This package guarantees strong cache consistency when using nds.Client.Get* and nds.Client.Put*, meaning you will never get data from a stale cache.

Exposed parts of this API are the same as the official one distributed by Google (code.google.com/go/datastore). However, underneath github.com/qedus/nds uses a caching stategy similar to the GAE Python NDB API. In fact the caching strategy used here even fixes one or two of the Python NDB caching consistency bugs.

You can find the API documentation at http://godoc.org/github.com/qedus/nds.

One other benefit is that the standard datastore.Client.GetMulti, datastore.Client.PutMulti and datastore.Client.DeleteMulti functions only allow you to work with a maximum of 1000, 500 and 500 entities per call respectively. The nds.Client.GetMulti, nds.Client.PutMulti and nds.Client.DeleteMulti functions in this package allow you to work with as many entities as you need (within timeout limits) by concurrently calling the appropriate datastore function until your request is fulfilled.

How To Use

You can use this package in exactly the same way you would use code.google.com/go/datastore.Client for methods provided by nds.Client. However, it is important that you use a nds.Client entirely within your code. Do not mix use of those functions with the code.google.com/go/datastore.Client equivalents as you will be liable to get stale datastore entities from github.com/qedus/nds.

Ultimately all you need to do is:

• import github.com/qedus/nds/v2
• use nds.NewClient instead of datastore.NewClient, providing a cache configuration to the new client creation function.
• replace datastore.Transaction -> nds.Transaction
• if using (*datastore.Query).Transaction for queries within transactions, switch to the (*nds.Transaction).Query helper.

Documentation

Overview

Package nds is a Go datastore API for Google Cloud Datastore that caches datastore calls in a cache in a strongly consistent manner. This often has the effect of making your app faster as cache access is often 10x faster than datastore access. It can also make your app cheaper to run as cache calls are typically cheaper.

This package goes to great lengths to ensure that stale datastore values are never returned to clients, i.e. the caching layer is strongly consistent. It does this by using a similar strategy to Python's ndb. However, this package fixes a couple of subtle edge case bugs that are found in ndb. See http://goo.gl/3ByVlA for one such bug.

There are currently no known consistency issues with the caching strategy employed by this package.

Use

Package nds' Client is used exactly the same way as the cloud.google.com/go/datastore.Client for implemented calls. Ensure that you change all your datastore client Get, Put, Delete, Mutate, and RunInTransaction function calls to use the nds client and Transaction type when converting your own code. The one caveat with transactions is when running queries, there is a helper function for adding the transaction to a datastore.Query.

If you mix datastore and nds API calls then you are liable to get stale cache.

Implement your own cache

You can implement your own nds.Cacher and use it in place of the cache backends provided by this package. The cache backends offered by Google such as AppEngine's Memcache and Cloud Memorystore (redis) are available via this package and can be used as references when adding your own.

Index

• Variables
• type Cacher
• type Client
  • func NewClient(ctx context.Context, cacher Cacher, opts ...ClientOption) (*Client, error)
  • func (c *Client) Delete(ctx context.Context, key *datastore.Key) error
  • func (c *Client) DeleteMulti(ctx context.Context, keys []*datastore.Key) error
  • func (c *Client) Get(ctx context.Context, key *datastore.Key, val interface{}) error
  • func (c *Client) GetMulti(ctx context.Context, keys []*datastore.Key, vals interface{}) error
  • func (c *Client) Mutate(ctx context.Context, muts ...*Mutation) ([]*datastore.Key, error)
  • func (c *Client) NewTransaction(ctx context.Context, opts ...datastore.TransactionOption) (t *Transaction, err error)
  • func (c *Client) Put(ctx context.Context, key *datastore.Key, val interface{}) (*datastore.Key, error)
  • func (c *Client) PutMulti(ctx context.Context, keys []*datastore.Key, vals interface{}) ([]*datastore.Key, error)
  • func (c *Client) RunInTransaction(ctx context.Context, f func(tx *Transaction) error, ...) (cmt *datastore.Commit, err error)
• type ClientOption
  • func WithDatastoreClient(ds *datastore.Client) ClientOption
  • func WithOnErrorFunc(f OnErrorFunc) ClientOption
• type Item
  • func (i *Item) GetCASInfo() interface{}
  • func (i *Item) SetCASInfo(value interface{})
• type MultiError
  • func (m MultiError) Error() string
• type Mutation
  • func NewDelete(k *datastore.Key) *Mutation
  • func NewInsert(k *datastore.Key, src interface{}) *Mutation
  • func NewUpdate(k *datastore.Key, src interface{}) *Mutation
  • func NewUpsert(k *datastore.Key, src interface{}) *Mutation
• type OnErrorFunc
• type Transaction
  • func (t *Transaction) Commit() (*datastore.Commit, error)
  • func (t *Transaction) Delete(key *datastore.Key) error
  • func (t *Transaction) DeleteMulti(keys []*datastore.Key) (err error)
  • func (t *Transaction) Get(key *datastore.Key, dst interface{}) error
  • func (t *Transaction) GetMulti(keys []*datastore.Key, dst interface{}) error
  • func (t *Transaction) Mutate(muts ...*Mutation) ([]*datastore.PendingKey, error)
  • func (t *Transaction) Put(key *datastore.Key, src interface{}) (*datastore.PendingKey, error)
  • func (t *Transaction) PutMulti(keys []*datastore.Key, src interface{}) (ret []*datastore.PendingKey, err error)
  • func (t *Transaction) Query(q *datastore.Query) *datastore.Query
  • func (t *Transaction) Rollback() (err error)

Variables

var (
	// ErrCacheMiss means an item was not found in cache
	ErrCacheMiss = errors.New("nds: cache miss")
	// ErrCASConflict means the cache item was modified between Get and CAS calls
	ErrCASConflict = errors.New("nds: cas conflict")
	// ErrNotStored means that an item was not stored due to a condition check failure (e.g. during an Add or CompareAndSwap call)
	ErrNotStored = errors.New("nds: not stored")
)

var (

	// Tag Keys
	KeyKind, _ = tag.NewKey("kind")

	// Views
	AllViews = []*view.View{
		{
			Name:        "nds/cache_hit",
			Description: "The number of cache hits",
			Measure:     mCacheHit,
			Aggregation: view.Sum(),
			TagKeys:     []tag.Key{KeyKind},
		},
		{
			Name:        "nds/cache_miss",
			Description: "The number of cache misses",
			Measure:     mCacheMiss,
			Aggregation: view.Sum(),
			TagKeys:     []tag.Key{KeyKind},
		},
	}
)

Types

type Cacher

type Cacher interface {
	// AddMulti adds each provided Item into the cache if and only if the key for the item is not
	// currently in use. For any item that was not stored due to a key conflict, a MultiError is returned
	// with ErrNotStored in the corresponding index for that item. There may be other errors for each MultiError index.
	AddMulti(ctx context.Context, items []*Item) error
	// CompareAndSwapMulti will set each item in the cache if the item was unchanged since it was last got.
	// The items are modified versions of items returned by the GetMulti call for this Cacher.
	// If any item was not able to be stored, a MultiError will be returned with ErrCASConflict if the value in
	// cache was changed since the GetMulti call, or ErrNotStored if the key was evicted or could otherwise not be
	// stored in the corresponding index for that item. There may be other errors for each MultiError index.
	CompareAndSwapMulti(ctx context.Context, items []*Item) error
	// DeleteMulti deletes all keys provided in the cache. It is not necessary to track individual key misses though
	// the proper way would be to return a MultiError with ErrCacheMiss stored in the corresponding index for that key.
	DeleteMulti(ctx context.Context, keys []string) error
	// GetMulti fetches all provided keys from the cache, returning a map[string]*Item filled with all Items
	// it was able to retrieve, using the item key as the lookup key. The cache implementation should utilize the SetCAS/
	// GetCAS info calls of each Item as required so subsequent calls to CompareAndSwapMulti can correctly execute. No
	// error should be returned for any key misses.
	GetMulti(ctx context.Context, keys []string) (map[string]*Item, error)
	// SetMulti sets all provided items in the cache, regardless of whether or not the key is already in use and the
	// value of that stored item. If any item could not be stored A MultiError should be returned with an error in the
	// corresponding index for that item. Note: ErrNotStored is not a valid error to be used for any error returned.
	SetMulti(ctx context.Context, items []*Item) error
}

Cacher represents a cache backend that can be used by nds.

type Client

type Client struct {

	// TODO: Client is exported since we embedded datastore.Client - fix this
	*datastore.Client
	// contains filtered or unexported fields
}

func NewClient

func NewClient(ctx context.Context, cacher Cacher, opts ...ClientOption) (*Client, error)

NewClient will return an nds.Client that can be used exactly like a datastore.Client but will transparently use the cache configuration provided to cache requests when it can.

func (*Client) Delete

func (c *Client) Delete(ctx context.Context, key *datastore.Key) error

Delete deletes the entity for the given key.

func (*Client) DeleteMulti

func (c *Client) DeleteMulti(ctx context.Context, keys []*datastore.Key) error

DeleteMulti works just like datastore.DeleteMulti except it maintains cache consistency with other NDS methods. It also removes the API limit of 500 entities per request by calling the datastore as many times as required to put all the keys. It does this efficiently and concurrently.

func (*Client) Get

func (c *Client) Get(ctx context.Context, key *datastore.Key, val interface{}) error

Get loads the entity stored for key into val, which must be a struct pointer. Currently PropertyLoadSaver and KeyLoader is implemented. If there is no such entity for the key, Get returns ErrNoSuchEntity.

The values of val's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each Get call.

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. ErrFieldMismatch is only returned if val is a struct pointer.

func (*Client) GetMulti

func (c *Client) GetMulti(ctx context.Context,
	keys []*datastore.Key, vals interface{}) error

GetMulti works similar to datastore.GetMulti except for two important advantages:

1) It removes the API limit of 1000 entities per request by calling the datastore as many times as required to fetch all the keys. It does this efficiently and concurrently.

2) GetMulti function will automatically use the cache where possible before accssing the datastore. It uses a caching mechanism similar to the Python ndb package. However consistency is improved as NDB consistency issue http://goo.gl/3ByVlA is not an issue here or accessing the same key concurrently.

If the cache is not working for any reason, GetMulti will default to using the datastore without compromising cache consistency.

Important: If you use nds.GetMulti, you must also use the NDS put and delete functions in all your code touching the datastore to ensure data consistency. This includes using nds.RunInTransaction instead of datastore.RunInTransaction.

Increase the datastore timeout if you get datastore_v3: TIMEOUT errors when getting thousands of entities. You can do this using context.WithTimeout https://golang.org/pkg/context/#WithTimeout.

vals must be a []S, []*S, []I or []P, for some struct type S, some interface type I, or some non-interface non-pointer type P such that P or *P implements datastore.PropertyLoadSaver. If an []I, each element must be a valid dst for Get: it must be a struct pointer or implement datastore.PropertyLoadSaver.

As a special case, datastore.PropertyList is an invalid type for dst, even though a PropertyList is a slice of structs. It is treated as invalid to avoid being mistakenly passed when []datastore.PropertyList was intended.

func (*Client) Mutate

func (c *Client) Mutate(ctx context.Context, muts ...*Mutation) ([]*datastore.Key, error)

func (*Client) NewTransaction

func (c *Client) NewTransaction(ctx context.Context, opts ...datastore.TransactionOption) (t *Transaction, err error)

NewTransaction will start a new datastore.Trnsaction wrapped by nds to properly update the cache

func (*Client) Put

func (c *Client) Put(ctx context.Context,
	key *datastore.Key, val interface{}) (*datastore.Key, error)

Put saves the entity val into the datastore with key. val must be a struct pointer; if a struct pointer then any unexported fields of that struct will be skipped. If key is an incomplete key, the returned key will be a unique key generated by the datastore.

func (*Client) PutMulti

func (c *Client) PutMulti(ctx context.Context,
	keys []*datastore.Key, vals interface{}) ([]*datastore.Key, error)

PutMulti is a batch version of Put. It works just like datastore.PutMulti except it interacts appropriately with NDS's caching strategy. It also removes the API limit of 500 entities per request by calling the datastore as many times as required to put all the keys. It does this efficiently and concurrently.

func (*Client) RunInTransaction

func (c *Client) RunInTransaction(ctx context.Context, f func(tx *Transaction) error, opts ...datastore.TransactionOption) (cmt *datastore.Commit, err error)

RunInTransaction works just like datastore.RunInTransaction however it interacts correctly with the cache. You should always use this method for transactions if you are using the NDS package.

type ClientOption

type ClientOption func(*Client)

ClientOption is an option for a nds Client. Inspired by Google's api/option package.

func WithDatastoreClient

func WithDatastoreClient(ds *datastore.Client) ClientOption

func WithOnErrorFunc

func WithOnErrorFunc(f OnErrorFunc) ClientOption

WithOnErrorFunc sets up an OnErrorFunc to be called for every internal error that doesn't return to the caller but maybe useful to capture for logging/debugging/reporting purposes. For example, to keep the original nds.v1 behavior of logging warnings in AppEngine Standard, this could be: func(ctx context.Context, err error) { log.Warningf(ctx, "%s", err) } By default this will log the error using the standard go log package.

type Item

type Item struct {
	// Key is the Item's key (250 bytes maximum).
	Key string
	// Value is the Item's value.
	Value []byte
	// Flags are server-opaque flags whose semantics are entirely up to the
	// app.
	Flags uint32
	// Expiration is the maximum duration that the item will stay
	// in the cache.
	// The zero value means the Item has no expiration time.
	// Subsecond precision is ignored.
	// This is not set when getting items.
	Expiration time.Duration
	// contains filtered or unexported fields
}

Item is the unit of Cacher gets and sets. Taken from google.golang.org/appengine/v2/memcache

func (*Item) GetCASInfo

func (i *Item) GetCASInfo() interface{}

GetCASInfo will return the value stored for this item for use with compare-and-swap operations.

func (*Item) SetCASInfo

func (i *Item) SetCASInfo(value interface{})

SetCASInfo is used by the Cacher implementation as needed for use with compare-and-swap operations. It will only set the value for the item once with subsequent calls being silently discarded.

type MultiError

type MultiError []error

MultiError is returned by batch operations when there are errors with particular elements. Errors will be in a one-to-one correspondence with the input elements; successful elements will have a nil entry.

func (MultiError) Error

func (m MultiError) Error() string

type Mutation

type Mutation struct {
	// contains filtered or unexported fields
}

func NewDelete

func NewDelete(k *datastore.Key) *Mutation

func NewInsert

func NewInsert(k *datastore.Key, src interface{}) *Mutation

func NewUpdate

func NewUpdate(k *datastore.Key, src interface{}) *Mutation

func NewUpsert

func NewUpsert(k *datastore.Key, src interface{}) *Mutation

type OnErrorFunc

type OnErrorFunc func(ctx context.Context, err error)

type Transaction

type Transaction struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func (*Transaction) Commit

func (t *Transaction) Commit() (*datastore.Commit, error)

Commit will commit the cache changes, then commit the transaction

func (*Transaction) Delete

func (t *Transaction) Delete(key *datastore.Key) error

func (*Transaction) DeleteMulti

func (t *Transaction) DeleteMulti(keys []*datastore.Key) (err error)

DeleteMulti is a batch version of Delete. It queues up all keys provided to be locked in the cache.

func (*Transaction) Get

func (t *Transaction) Get(key *datastore.Key, dst interface{}) error

func (*Transaction) GetMulti

func (t *Transaction) GetMulti(keys []*datastore.Key, dst interface{}) error

GetMulti is a batch version of Get. It bypasses the cache during transactions.

func (*Transaction) Mutate

func (t *Transaction) Mutate(muts ...*Mutation) ([]*datastore.PendingKey, error)

Mutate will lock all keys from the mutations provided.

func (*Transaction) Put

func (t *Transaction) Put(key *datastore.Key, src interface{}) (*datastore.PendingKey, error)

func (*Transaction) PutMulti

func (t *Transaction) PutMulti(keys []*datastore.Key, src interface{}) (ret []*datastore.PendingKey, err error)

PutMulti in a batch version of Put. It queues up all keys provided to be locked in the cache.

func (*Transaction) Query

func (t *Transaction) Query(q *datastore.Query) *datastore.Query

Query is a helper function to use underlying *datastore.Transaction for queries in nds Transactions

func (*Transaction) Rollback

func (t *Transaction) Rollback() (err error)

Rollback is just a passthrough to the underlying datastore.Transaction.