goon

README

goon

An autocaching interface to the app engine datastore for Go. Designed to be similar to the python NDB package.

Documentation: http://godoc.org/github.com/mjibson/goon

Documentation

Overview

Package goon provides an autocaching interface to the app engine datastore similar to the python NDB package.

Goon differs from the datastore package in various ways: it remembers the appengine Context, which need only be specified once at creation time; kinds need not be specified as they are computed, by default, from a type's name; keys are inferred from specially-tagged fields on types, removing the need to pass key objects around.

In general, the difference is that Goon's API is identical to the datastore API, it's just shorter.

Keys in Goon are stored in the structs themselves. Below is an example struct with a field to specify the id (see the Key Specifications section below for full documentation).

type User struct {
	Id    string `datastore:"-" goon:"id"`
	Name  string
}

Thus, to get a User with id 2:

userid := 2
g := goon.NewGoon(r)
u := &User{Id: userid}
g.Get(u)

Key Specifications

For both the Key and KeyError functions, src must be a S or *S for some struct type S. The key is extracted based on various fields of S. If a field of type int64 or string has a struct tag named goon with value "id", it is used as the key's id. If a field of type *datastore.Key has a struct tag named goon with value "parent", it is used as the key's parent. If a field of type string has a struct tag named goon with value "kind", it is used as the key's kind. The "kind" field supports an optional second parameter which is the default kind name. If no kind field exists, the struct's name is used. These fields should all have their datastore field marked as "-".

Example, with kind User:

type User struct {
	Id    string `datastore:"-" goon:"id"`
	Read  time.Time
}

Example, with kind U if _kind is the empty string:

type User struct {
	_kind string `goon:"kind,U"`
	Id    string `datastore:"-" goon:"id"`
	Read  time.Time
}

To override kind of a single entity to UserKind:

u := User{_kind: "UserKind"}

An example with both parent and kind:

type UserData struct {
	Id     string         `datastore:"-" goon:"id"`
	_kind  string         `goon:"kind,UD"`
	Parent *datastore.Key `datastore:"-" goon:"parent"`
	Data   []byte
}

Features

Datastore interaction with: Get, GetMulti, Put, PutMulti, Delete, DeleteMulti, Queries.

All key-based operations backed by memory and memcache.

Per-request, in-memory cache: fetch the same key twice, the second request is served from local memory.

Intelligent multi support: running GetMulti correctly fetches from memory, then memcache, then the datastore; each tier only sends keys off to the next one if they were missing.

Memcache control variance: long memcache requests are cancelled.

Transactions use a separate context, but locally cache any results on success.

Automatic kind naming: struct names are inferred by reflection, removing the need to manually specify key kinds.

Simpler API than appengine/datastore.

API comparison between goon and datastore

put with incomplete key

datastore:

type Group struct {
	Name string
}
c := appengine.NewContext(r)
g := &Group{Name: "name"}
k := datastore.NewIncompleteKey(c, "Group", nil)
err := datastore.Put(c, k, g)

goon:

type Group struct {
	Id   int64 `datastore:"-" goon:"id"`
	Name string
}
n := goon.NewGoon(r)
g := &Group{Name: "name"}
err := n.Put(g)

get with known key

datastore:

type Group struct {
	Name string
}
c := appengine.NewContext(r)
g := &Group{}
k := datastore.NewKey(c, "Group", "", 1, nil)
err := datastore.Get(c, k, g)

goon:

type Group struct {
	Id   int64 `datastore:"-" goon:"id"`
	Name string
}
n := goon.NewGoon(r)
g := &Group{Id: 1}
err := n.Get(g)

Memcache Control Variance

Memcache is generally fast. When it is slow, goon will timeout the memcache requests and proceed to use the datastore directly. The memcache put and get timeout variables determine how long to wait for various kinds of requests. The default settings were determined experimentally and should provide reasonable defaults for most applications.

See: http://talks.golang.org/2013/highperf.slide#23

Index

• Variables
• func DefaultKindName(src interface{}) string
• func NotFound(err error, idx int) bool
• type Goon
  • func FromContext(c context.Context) *Goon
  • func NewGoon(r *http.Request) *Goon
  • func (g *Goon) Count(q *datastore.Query) (int, error)
  • func (g *Goon) Delete(key *datastore.Key) error
  • func (g *Goon) DeleteMulti(keys []*datastore.Key) error
  • func (g *Goon) FlushLocalCache()
  • func (g *Goon) Get(dst interface{}) error
  • func (g *Goon) GetAll(q *datastore.Query, dst interface{}) ([]*datastore.Key, error)
  • func (g *Goon) GetMulti(dst interface{}) error
  • func (g *Goon) Key(src interface{}) *datastore.Key
  • func (g *Goon) KeyError(src interface{}) (*datastore.Key, error)
  • func (g *Goon) Kind(src interface{}) string
  • func (g *Goon) Put(src interface{}) (*datastore.Key, error)
  • func (g *Goon) PutMulti(src interface{}) ([]*datastore.Key, error)
  • func (g *Goon) Run(q *datastore.Query) *Iterator
  • func (g *Goon) RunInTransaction(f func(tg *Goon) error, opts *datastore.TransactionOptions) error
• type Iterator
  • func (t *Iterator) Cursor() (datastore.Cursor, error)
  • func (t *Iterator) Next(dst interface{}) (*datastore.Key, error)
• type KindNameResolver

Variables

var (
	// LogErrors issues appengine.Context.Errorf on any error.
	LogErrors = true
	// LogTimeoutErrors issues appengine.Context.Warningf on memcache timeout errors.
	LogTimeoutErrors = false

	// MemcachePutTimeoutThreshold is the number of bytes at which the memcache
	// timeout uses the large setting.
	MemcachePutTimeoutThreshold = 1024 * 50
	// MemcachePutTimeoutSmall is the amount of time to wait during memcache
	// Put operations before aborting them and using the datastore.
	MemcachePutTimeoutSmall = time.Millisecond * 5
	// MemcachePutTimeoutLarge is the amount of time to wait for large memcache
	// Put requests.
	MemcachePutTimeoutLarge = time.Millisecond * 15
	// MemcacheGetTimeout is the amount of time to wait for all memcache Get
	// requests.
	MemcacheGetTimeout = time.Millisecond * 10

	// IgnoreFieldMismatch decides whether *datastore.ErrFieldMismatch errors
	// should be silently ignored. This allows you to easily remove fields from structs.
	IgnoreFieldMismatch = true
)

var MemcacheKey = func(k *datastore.Key) string {

	return "g2:" + k.Encode()
}

MemcacheKey returns key string of Memcache.

Functions

func DefaultKindName

func DefaultKindName(src interface{}) string

DefaultKindName is the default implementation to determine the Kind an Entity has. Returns the basic Type of the src (no package name included).

func NotFound

func NotFound(err error, idx int) bool

NotFound returns true if err is an appengine.MultiError and err[idx] is a datastore.ErrNoSuchEntity.

Types

type Goon

type Goon struct {
	Context context.Context

	// KindNameResolver is used to determine what Kind to give an Entity.
	// Defaults to DefaultKindName
	KindNameResolver KindNameResolver
	// contains filtered or unexported fields
}

Goon holds the app engine context and the request memory cache.

func FromContext

func FromContext(c context.Context) *Goon

FromContext creates a new Goon object from the given appengine Context. Useful with profiling packages like appstats.

func NewGoon

func NewGoon(r *http.Request) *Goon

NewGoon creates a new Goon object from the given request.

func (*Goon) Count

func (g *Goon) Count(q *datastore.Query) (int, error)

Count returns the number of results for the query.

func (*Goon) Delete

func (g *Goon) Delete(key *datastore.Key) error

Delete deletes the entity for the given key.

func (*Goon) DeleteMulti

func (g *Goon) DeleteMulti(keys []*datastore.Key) error

DeleteMulti is a batch version of Delete.

func (*Goon) FlushLocalCache

func (g *Goon) FlushLocalCache()

FlushLocalCache clears the local memory cache.

func (*Goon) Get

func (g *Goon) Get(dst interface{}) error

Get loads the entity based on dst's key into dst If there is no such entity for the key, Get returns datastore.ErrNoSuchEntity.

func (*Goon) GetAll

func (g *Goon) GetAll(q *datastore.Query, dst interface{}) ([]*datastore.Key, error)

GetAll runs the query and returns all the keys that match the query, as well as appending the values to dst, setting the goon key fields of dst, and caching the returned data in local memory.

For "keys-only" queries dst can be nil, however if it is not, then GetAll appends zero value structs to dst, only setting the goon key fields. No data is cached with "keys-only" queries.

See: https://developers.google.com/appengine/docs/go/datastore/reference#Query.GetAll

func (*Goon) GetMulti

func (g *Goon) GetMulti(dst interface{}) error

GetMulti is a batch version of Get.

dst must be a *[]S, *[]*S, *[]I, []S, []*S, or []I, for some struct type S, or some interface type I. If *[]I or []I, each element must be a struct pointer.

func (*Goon) Key

func (g *Goon) Key(src interface{}) *datastore.Key

Key is the same as KeyError, except nil is returned on error or if the key is incomplete.

func (*Goon) KeyError

func (g *Goon) KeyError(src interface{}) (*datastore.Key, error)

KeyError returns the key of src based on its properties.

func (*Goon) Kind

func (g *Goon) Kind(src interface{}) string

Kind returns src's datastore Kind or "" on error.

func (*Goon) Put

func (g *Goon) Put(src interface{}) (*datastore.Key, error)

Put saves the entity src into the datastore based on src's key k. If k is an incomplete key, the returned key will be a unique key generated by the datastore.

func (*Goon) PutMulti

func (g *Goon) PutMulti(src interface{}) ([]*datastore.Key, error)

PutMulti is a batch version of Put.

src must be a *[]S, *[]*S, *[]I, []S, []*S, or []I, for some struct type S, or some interface type I. If *[]I or []I, each element must be a struct pointer.

func (*Goon) Run

func (g *Goon) Run(q *datastore.Query) *Iterator

Run runs the query.

func (*Goon) RunInTransaction

func (g *Goon) RunInTransaction(f func(tg *Goon) error, opts *datastore.TransactionOptions) error

RunInTransaction runs f in a transaction. It calls f with a transaction context tg that f should use for all App Engine operations. Neither cache nor memcache are used or set during a transaction.

Otherwise similar to appengine/datastore.RunInTransaction: https://developers.google.com/appengine/docs/go/datastore/reference#RunInTransaction

type Iterator

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator is the result of running a query.

func (*Iterator) Cursor

func (t *Iterator) Cursor() (datastore.Cursor, error)

Cursor returns a cursor for the iterator's current location.

func (*Iterator) Next

func (t *Iterator) Next(dst interface{}) (*datastore.Key, error)

Next returns the entity of the next result. When there are no more results, datastore.Done is returned as the error. If dst is null (for a keys-only query), nil is returned as the entity.

If the query is not keys only and dst is non-nil, it also loads the entity stored for that key into the struct pointer dst, with the same semantics and possible errors as for the Get function. This result is cached in memory.

If the query is keys only, dst must be passed as nil. Otherwise the cache will be populated with empty entities since there is no way to detect the case of a keys-only query.

Refer to appengine/datastore.Iterator.Next: https://developers.google.com/appengine/docs/go/datastore/reference#Iterator.Next

type KindNameResolver

type KindNameResolver func(src interface{}) string

KindNameResolver takes an Entity and returns what the Kind should be for Datastore.