nds

README

nds

Package github.com/qedus/nds is a datastore API for the Google App Engine (GAE) Go Runtime Environment that uses memcache to cache all datastore requests. It is compatible with both Classic and Managed VM products. This package guarantees strong cache consistency when using nds.Get* and nds.Put*, meaning you will never get data from a stale cache.

Exposed parts of this API are the same as the official one distributed by Google (google.golang.org/appengine/datastore). However, underneath github.com/qedus/nds uses a caching stategy similar to the GAE Python NDB API. In fact the caching strategy used here even fixes one or two of the Python NDB caching consistency bugs.

You can find the API documentation at http://godoc.org/github.com/qedus/nds.

One other benefit is that the standard datastore.GetMulti, datastore.PutMulti and datastore.DeleteMulti functions only allow you to work with a maximum of 1000, 500 and 500 entities per call respectively. The nds.GetMulti, nds.PutMulti and nds.DeleteMulti functions in this package allow you to work with as many entities as you need (within timeout limits) by concurrently calling the appropriate datastore function until your request is fulfilled.

How To Use

You can use this package in exactly the same way you would use google.golang.org/appengine/datastore. However, it is important that you use nds.Get*, nds.Put*, nds.Delete* and nds.RunInTransaction entirely within your code. Do not mix use of those functions with the google.golang.org/appengine/datastore equivalents as you will be liable to get stale datastore entities from github.com/qedus/nds.

Ultimately all you need to do is find/replace the following in your codebase:

• datastore.Get -> nds.Get
• datastore.Put -> nds.Put
• datastore.Delete -> nds.Delete
• datastore.RunInTransaction -> nds.RunInTransaction

Versions

Versions are specified using Go Modules.

• Version 1.x.x: Can be found on branch master and uses the google.golang.org/appengine package.
• Version 2.x.x: Can be found on branch v2. This is a major update and takes advantage of the new cloud.google.com/go/datastore package provided by Google. It currently in an experimental state and we welcome contributions.

Documentation

Overview

Package nds is a Go datastore API for Google App Engine that caches datastore calls in memcache in a strongly consistent manner. This often has the effect of making your app faster as memcache access is often 10x faster than datastore access. It can also make your app cheaper to run as memcache calls are free.

This package goes to great lengths to ensure that stale datastore values are never returned to clients, i.e. the caching layer is strongly consistent. It does this by using a similar strategy to Python's ndb. However, this package fixes a couple of subtle edge case bugs that are found in ndb. See http://goo.gl/3ByVlA for one such bug.

There are currently no known consistency issues with the caching strategy employed by this package.

Use

Package nds is used exactly the same way as appeninge/datastore. Ensure that you change all your datastore Get, Put, Delete and RunInTransaction function calls to use nds when converting your own code.

If you mix appengine/datastore and nds API calls then you are liable to get stale cache.

Converting Legacy Code

To convert legacy code you will need to find and replace all invocations of datastore.Get, datastore.Put, datastore.Delete, datastore.RunInTransaction with nds.Get, nds.Put, nds.Delete and nds.RunInTransaction respectively.

Index

• func Delete(c context.Context, key *datastore.Key) error
• func DeleteMulti(c context.Context, keys []*datastore.Key) error
• func Get(c context.Context, key *datastore.Key, val interface{}) error
• func GetMulti(c context.Context, keys []*datastore.Key, vals interface{}) error
• func Put(c context.Context, key *datastore.Key, val interface{}) (*datastore.Key, error)
• func PutMulti(c context.Context, keys []*datastore.Key, vals interface{}) ([]*datastore.Key, error)
• func RunInTransaction(c context.Context, f func(tc context.Context) error, ...) error

Functions

func Delete

func Delete(c context.Context, key *datastore.Key) error

Delete deletes the entity for the given key.

func DeleteMulti

func DeleteMulti(c context.Context, keys []*datastore.Key) error

DeleteMulti works just like datastore.DeleteMulti except it maintains cache consistency with other NDS methods. It also removes the API limit of 500 entities per request by calling the datastore as many times as required to put all the keys. It does this efficiently and concurrently.

func Get

func Get(c context.Context, key *datastore.Key, val interface{}) error

Get loads the entity stored for key into val, which must be a struct pointer. Currently PropertyLoadSaver is not implemented. If there is no such entity for the key, Get returns ErrNoSuchEntity.

The values of val's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each Get call.

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. ErrFieldMismatch is only returned if val is a struct pointer.

func GetMulti

func GetMulti(c context.Context,
	keys []*datastore.Key, vals interface{}) error

GetMulti works similar to datastore.GetMulti except for two important advantages:

1) It removes the API limit of 1000 entities per request by calling the datastore as many times as required to fetch all the keys. It does this efficiently and concurrently.

2) GetMulti function will automatically use memcache where possible before accssing the datastore. It uses a caching mechanism similar to the Python ndb package. However consistency is improved as NDB consistency issue http://goo.gl/3ByVlA is not an issue here or accessing the same key concurrently.

If memcache is not working for any reason, GetMulti will default to using the datastore without compromising cache consistency.

Important: If you use nds.GetMulti, you must also use the NDS put and delete functions in all your code touching the datastore to ensure data consistency. This includes using nds.RunInTransaction instead of datastore.RunInTransaction.

Increase the datastore timeout if you get datastore_v3: TIMEOUT errors when getting thousands of entities. You can do this using http://godoc.org/code.google.com/p/appengine-go/appengine#Timeout.

vals must be a []S, []*S, []I or []P, for some struct type S, some interface type I, or some non-interface non-pointer type P such that P or *P implements datastore.PropertyLoadSaver. If an []I, each element must be a valid dst for Get: it must be a struct pointer or implement datastore.PropertyLoadSaver.

As a special case, datastore.PropertyList is an invalid type for dst, even though a PropertyList is a slice of structs. It is treated as invalid to avoid being mistakenly passed when []datastore.PropertyList was intended.

func Put

func Put(c context.Context,
	key *datastore.Key, val interface{}) (*datastore.Key, error)

Put saves the entity val into the datastore with key. val must be a struct pointer; if a struct pointer then any unexported fields of that struct will be skipped. If key is an incomplete key, the returned key will be a unique key generated by the datastore.

func PutMulti

func PutMulti(c context.Context,
	keys []*datastore.Key, vals interface{}) ([]*datastore.Key, error)

PutMulti is a batch version of Put. It works just like datastore.PutMulti except it interacts appropriately with NDS's caching strategy. It also removes the API limit of 500 entities per request by calling the datastore as many times as required to put all the keys. It does this efficiently and concurrently.

func RunInTransaction

func RunInTransaction(c context.Context, f func(tc context.Context) error,
	opts *datastore.TransactionOptions) error

RunInTransaction works just like datastore.RunInTransaction however it interacts correctly with memcache. You should always use this method for transactions if you are using the NDS package.