datastore

README

Datastore Wrapper

(AppEngine | Cloud) Datastore wrapper for Go 👉

Simple. Happy. Respect standard library.

$ go get -u go.mercari.io/datastore

Feature

DO

• Wrap google.golang.org/appengine/datastore and cloud.google.com/go/datastore
  • keep key behavior
  • align to cloud.google.com/go/datastore first
• Re-implement datastore package
• Re-implement datastore.SaveStruct & LoadStruct
  • Ignore unmapped property
  • Add PropertyTranslator interface
    • Convert types like mytime.Unix to time.Time and reverse it
    • Rename property like CreatedAt to createdAt or created_at and reverse it
• Re-implement PropertyLoadSaver
  • Pass context.Context to Save & Load method
• Add retry feature to each RPC
  • e.g. Retry AllocateID when it failed
• Add middleware layer
  • About...
    • Local Cache
    • AE Memcache
    • Logging
    • Retry
    • etc...
  • Easy to ON/OFF switching
• Add some useful methods
  • aedatastore/TransactionContext

DON'T

• have utility functions
• support firestore

Restriction

• aedatastore package
  • When using slice of struct, MUST specified datastore:",flatten" option.
    • original (ae & cloud) datastore.SaveStruct have different behaviors.
    • see aeprodtest/main.go /api/test3

Committers

• Masahiro Wakame (@vvakame)

Contribution

Please read the CLA below carefully before submitting your contribution.

https://www.mercari.com/cla/

Setup environment & Run tests

• requirements
  • gcloud sdk
    • gcloud components install app-engine-go
    • gcloud components install beta cloud-datastore-emulator

1. Testing in local

$ ./setup.sh # exec once
$ ./serve.sh # exec in background
$ ./test.sh

1. Testing with Circle CI CLI

$ circleci build

License

Copyright 2017 Mercari, Inc.

Licensed under the MIT License.

Documentation

Overview

Package datastore has an abstract representation of (AppEngine | Cloud) Datastore.

repository https://github.com/mercari/datastore

Let's read https://cloud.google.com/datastore/docs/ or https://cloud.google.com/appengine/docs/standard/go/datastore/ . You should also check https://godoc.org/cloud.google.com/go/datastore or https://godoc.org/google.golang.org/appengine/datastore as datastore original library.

Japanese version https://github.com/mercari/datastore/blob/master/doc_ja.go

Basic usage

Please see https://godoc.org/go.mercari.io/datastore/clouddatastore or https://godoc.org/go.mercari.io/datastore/aedatastore . Create a Client using the FromContext function of each package.

Later in this document, notes on migration from each package are summarized. Please see also there.

This package is based on the newly designed Cloud Datastore API. We are introducing flatten tags that only exist in Cloud Datastore, we need to be careful when migrating from AE Datastore. Details will be described later. If you are worried, you may have a clue to the solution at https://godoc.org/go.mercari.io/datastore/clouddatastore .

The purpose of this package

This package has three main objectives.

1. Provide a middleware layer, and reduce the code that are not directly related to application value.
2. AppEngine, Cloud, to provide the same interface for both Datastore.
3. Enable batch processing for Single Get, Signle Put, etc.

Middleware layer

We are forced to make functions that are not directly related to the value of the application for speed, stability and operation. Such functions can be abstracted and used as middleware.

Let's think about this case. Put Entity to Datastore and set it to Memcache or Redis. Next, when getting from Datastore, Get from Memcache first, Get it again from Datastore if it fails. It is very troublesome to provide these operations for all Kind and all Entity operations. However, if the middleware intervenes with all Datastore RPCs, you can transparently process without affecting the application code.

As another case, RPC sometimes fails. If it fails, the process often succeeds simply by retrying. For easy RET retry with all RPCs, it is better to implement it as middleware.

Please refer to https://godoc.org/go.mercari.io/datastore/dsmiddleware if you want to know the middleware already provided.

Provide the same interface between AppEngine and Cloud Datastore

The same interface is provided for AppEngine Datastore and Cloud Datastore. These two are compatible, you can run it with exactly the same code after creating the Client.

For example, you can use AE Datastore in a production environment and Cloud Datastore Emulator in UnitTest. If you can avoid goapp, tests may be faster and IDE may be more vulnerable to debugging. You can also read data from the local environment via Cloud Datastore for systems running on AE Datastore.

Caution. Although the storage bodies of RPCs of AE Datastore and Cloud Datastore are shared, there is a difference in expressiveness at the API level. Please carefully read the data written in AE Datastore carelessly on Cloud Datastore and do not update it. It may become impossible to read from the API of AE Datastore side. About this, we have not strictly tested.

Batch processing

The operation of Datastore has very little latency with respect to RPC's network. When acquiring 10 entities it means that GetMulti one time is better than getting 10 times using loops. However, we are not good at putting together multiple processes at once. Suppose, for example, you want to query on Post Kind, use the list of Comment IDs of the resulting Post, and get a list of Comments. For example, you can query Post Kind and get a list of Post. In addition, consider using CommentIDs of Post and getting a list of Comment. This is enough Query + 1 GetMulti is enough if you write very clever code. However, after acquiring the data, it is necessary to link the Comment list with the appropriate Post. On the other hand, you can easily write a code that throws a query once and then GetMulti the Comment as many as Post. In summary, it is convenient to have Put or Get queued, and there is a mechanism to execute it collectively later.

Batch() is it! You can find the example at https://godoc.org/go.mercari.io/datastore/#pkg-examples .

Boom replacing goon

I love goon. So I made https://godoc.org/go.mercari.io/datastore/boom which can be used in conjunction with this package.

How to migrate to this library

Here's an overview of what you need to do to migrate your existing code.

replace *datastore.Key to datastore.Key.
replace *datastore.Query to datastore.Query.
replace *datastore.Iterator to datastore.Iterator.

from AE Datastore

import go.mercari.io/datastore and go.mercari.io/datastore/aedatastore both.
rewrite those using functions of datastore package to FromContext function and Client method calls.
replace err.(appengine.MultiError) to err.(datastore.MultiError) .
Stop using appengine.BlobKey and replace with string.
replace google.golang.org/appengine/datastore.Done to google.golang.org/api/iterator.Done .
replace key.IntID() to key.ID() .
replace key.StringID() to key.Name() .
When nesting a struct, apply `datastore:", flatten "` to the corresponding field.
Delete datastore.TransactionOptions, it is not supported.
If using google.golang.org/appengine/datastore , replace to go.mercari.io/datastore .

from Cloud Datastore

import go.mercari.io/datastore and go.mercari.io/datastore/clouddatastore .
rewrite those using functions of datastore package to FromContext function and Client method calls.
replace *datastore.Commit to datastore.Commit .
If using cloud.google.com/go/datastore , replace to go.mercari.io/datastore .

from goon to boom

replace *goon.Goon to *boom.Boom .
replace goon.FromContext(ctx) to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds) .

Example (Batch)

package main

import (
	"context"
	"fmt"

	"go.mercari.io/datastore/clouddatastore"
	"go.mercari.io/datastore/internal/testutils"
)

func main() {
	ctx := context.Background()
	client, err := clouddatastore.FromContext(ctx)
	if err != nil {
		panic(err)
	}
	defer client.Close()
	defer testutils.CleanUpAllEntities(ctx, client)

	type Comment struct {
		Message string
	}
	type Post struct {
		Content    string
		CommentIDs []int64    `json:"-"`
		Comments   []*Comment `datastore:"-"`
	}

	// preparing entities
	for i := 0; i < 4; i++ {
		post := &Post{Content: fmt.Sprintf("post #%d", i+1)}
		key, err := client.Put(ctx, client.IncompleteKey("Post", nil), post)
		if err != nil {
			panic(err)
		}

		for j := 0; j < 5; j++ {
			comment := &Comment{Message: fmt.Sprintf("comment #%d", j+1)}
			cKey, err := client.Put(ctx, client.IncompleteKey("Comment", nil), comment)
			if err != nil {
				panic(err)
			}

			post.CommentIDs = append(post.CommentIDs, cKey.ID())
		}
		_, err = client.Put(ctx, key, post)
		if err != nil {
			panic(err)
		}
	}

	// start fetching...
	posts := make([]*Post, 0)
	_, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts)
	if err != nil {
		panic(err)
	}

	// Let's batch get!
	bt := client.Batch()

	for _, post := range posts {
		comments := make([]*Comment, 0)
		for _, id := range post.CommentIDs {
			comment := &Comment{}
			bt.Get(client.IDKey("Comment", id, nil), comment, nil)
			comments = append(comments, comment)
		}
		post.Comments = comments
	}

	err = bt.Exec(ctx)
	if err != nil {
		panic(err)
	}

	// check result
	for _, post := range posts {
		fmt.Println("Post", post.Content)
		for _, comment := range post.Comments {
			fmt.Println("Comment", comment.Message)
		}
	}

}

Output:

Post post #1
Comment comment #1
Comment comment #2
Comment comment #3
Comment comment #4
Comment comment #5
Post post #2
Comment comment #1
Comment comment #2
Comment comment #3
Comment comment #4
Comment comment #5
Post post #3
Comment comment #1
Comment comment #2
Comment comment #3
Comment comment #4
Comment comment #5
Post post #4
Comment comment #1
Comment comment #2
Comment comment #3
Comment comment #4
Comment comment #5

Example (BatchWithBatchErrHandler)

package main

import (
	"context"
	"fmt"

	"go.mercari.io/datastore"
	"go.mercari.io/datastore/clouddatastore"
	"go.mercari.io/datastore/internal/testutils"
)

func main() {
	ctx := context.Background()
	client, err := clouddatastore.FromContext(ctx)
	if err != nil {
		panic(err)
	}
	defer client.Close()
	defer testutils.CleanUpAllEntities(ctx, client)

	type Comment struct {
		Message string
	}

	// preparing entities...
	// Put ID: 2, 4 into Datastore.
	var keys []datastore.Key
	for i := 1; i <= 5; i++ {
		key := client.IDKey("Comment", int64(i), nil)
		keys = append(keys, key)

		comment := &Comment{Message: fmt.Sprintf("comment #%d", i)}
		if i%2 == 0 {
			_, err = client.Put(ctx, key, comment)
			if err != nil {
				panic(err)
			}
		}
	}

	// Let's batch get!
	bt := client.Batch()

	var comments []*Comment
	for _, key := range keys {
		comment := &Comment{}

		bt.Get(key, comment, func(err error) error {
			if err == datastore.ErrNoSuchEntity {
				// ignore ErrNoSuchEntity
				return nil
			} else if err != nil {
				return err
			}

			comments = append(comments, comment)

			return nil
		})
	}

	err = bt.Exec(ctx)
	if err != nil {
		panic(err)
	}

	// check result
	for _, comment := range comments {
		fmt.Println("Comment", comment.Message)
	}

}

Output:

Comment comment #2
Comment comment #4

Example (ClientGet)

package main

import (
	"context"
	"fmt"

	"go.mercari.io/datastore/clouddatastore"
	"go.mercari.io/datastore/internal/testutils"
)

func main() {
	ctx := context.Background()
	client, err := clouddatastore.FromContext(ctx)
	if err != nil {
		panic(err)
	}
	defer client.Close()
	defer testutils.CleanUpAllEntities(ctx, client)

	type Data struct {
		Name string
	}

	key := client.IncompleteKey("Data", nil)
	entity := &Data{Name: "mercari"}
	key, err = client.Put(ctx, key, entity)
	if err != nil {
		panic(err)
	}

	entity = &Data{}
	err = client.Get(ctx, key, entity)
	if err != nil {
		panic(err)
	}

	fmt.Println(entity.Name)
}

Output:

mercari

Index

• Variables
• func LoadStruct(ctx context.Context, dst interface{}, p []Property) error
• type Batch
  • func (b *Batch) Delete(key Key, h BatchErrHandler)
  • func (b *Batch) Exec(ctx context.Context) error
  • func (b *Batch) Get(key Key, dst interface{}, h BatchErrHandler)
  • func (b *Batch) Put(key Key, src interface{}, h BatchPutHandler)
• type BatchErrHandler
• type BatchPutHandler
• type Client
• type ClientGenerator
• type ClientOption
  • func WithCredentialsFile(filename string) ClientOption
  • func WithGRPCDialOption(opt grpc.DialOption) ClientOption
  • func WithHTTPClient(client *http.Client) ClientOption
  • func WithProjectID(projectID string) ClientOption
  • func WithScopes(scope ...string) ClientOption
  • func WithTokenSource(s oauth2.TokenSource) ClientOption
• type Commit
• type Cursor
• type Entity
• type ErrFieldMismatch
  • func (e *ErrFieldMismatch) Error() string
• type GeoPoint
• type Iterator
• type Key
• type KeyLoader
• type Middleware
• type MiddlewareInfo
• type MultiError
  • func (m MultiError) Error() string
• type PendingKey
• type Property
  • func SaveStruct(ctx context.Context, src interface{}) ([]Property, error)
• type PropertyList
  • func (l *PropertyList) Load(ctx context.Context, p []Property) error
  • func (l *PropertyList) Save(ctx context.Context) ([]Property, error)
• type PropertyLoadSaver
• type PropertyTranslator
• type Query
• type QueryDump
  • func (dump *QueryDump) String() string
• type QueryFilterCondition
• type Transaction
• type TransactionBatch
  • func (b *TransactionBatch) Delete(key Key, h BatchErrHandler)
  • func (b *TransactionBatch) Exec() error
  • func (b *TransactionBatch) Get(key Key, dst interface{}, h BatchErrHandler)
  • func (b *TransactionBatch) Put(key Key, src interface{}, h TxBatchPutHandler)
• type TxBatchPutHandler

Examples

• Package (Batch)
• Package (BatchWithBatchErrHandler)
• Package (ClientGet)
• PropertyTranslator

Variables

var (
	// ErrInvalidEntityType is returned when functions like Get or Next are
	// passed a dst or src argument of invalid type.
	ErrInvalidEntityType = errors.New("datastore: invalid entity type")
	// ErrInvalidKey is returned when an invalid key is presented.
	ErrInvalidKey = errors.New("datastore: invalid key")
	// ErrNoSuchEntity is returned when no entity was found for a given key.
	ErrNoSuchEntity = errors.New("datastore: no such entity")
)

var ErrConcurrentTransaction = errors.New("datastore: concurrent transaction")

ErrConcurrentTransaction is returned when a transaction is rolled back due to a conflict with a concurrent transaction.

var LoadEntity = loadEntity

LoadEntity to dst struct.

var SaveEntity = saveEntity

SaveEntity convert key & struct to *Entity.

var SuppressErrFieldMismatch = true

SuppressErrFieldMismatch when this flag is true. If you want to align (AE|Cloud) Datastore's default behavior, set false.

Functions

func LoadStruct

func LoadStruct(ctx context.Context, dst interface{}, p []Property) error

LoadStruct loads the properties from p to dst. dst must be a struct pointer.

The values of dst's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each LoadStruct call.

Types

type Batch

type Batch struct {
	Client Client
	// contains filtered or unexported fields
}

Batch can queue operations on Datastore and process them in batch. Batch does nothing until you call Exec(). This helps to reduce the number of RPCs.

func (*Batch) Delete

func (b *Batch) Delete(key Key, h BatchErrHandler)

Delete Entity operation into the queue.

func (*Batch) Exec

func (b *Batch) Exec(ctx context.Context) error

Exec will perform all the processing that was queued. This process is done recursively until the queue is empty. The return value may be MultiError, but the order of contents is not guaranteed.

func (*Batch) Get

func (b *Batch) Get(key Key, dst interface{}, h BatchErrHandler)

Get Entity operation into the queue.

func (*Batch) Put

func (b *Batch) Put(key Key, src interface{}, h BatchPutHandler)

Put Entity operation into the queue. This operation doesn't Put to Datastore immediately. If a h is provided, it passes the processing result to the handler, and treats the return value as the value of the result of Putting.

type BatchErrHandler

type BatchErrHandler func(err error) error

BatchErrHandler represents Entity's individual callback when batching non-Put processing.

type BatchPutHandler

type BatchPutHandler func(key Key, err error) error

BatchPutHandler represents Entity's individual callback when batching Put processing.

type Client

type Client interface {
	// Get loads the entity stored for key into dst, which must be a struct pointer or implement PropertyLoadSaver.
	//
	// If there is no such entity for the key, Get returns ErrNoSuchEntity.
	// The values of dst's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them.
	// In particular, it is recommended to pass a pointer to a zero valued struct on each Get call.
	//
	// If you set false to SuppressErrFieldMismatch variable, act like the original Datastore.
	// ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct.
	Get(ctx context.Context, key Key, dst interface{}) error

	// GetMulti is a batch version of Get.
	//
	// dst must be a []S, []*S, []I or []P, for some struct type S, some interface type I, or some non-interface non-pointer type P such that P or *P implements PropertyLoadSaver.
	// If an []I, each element must be a valid dst for Get: it must be a struct pointer or implement PropertyLoadSaver.
	//
	// As a special case, PropertyList is an invalid type for dst, even though a PropertyList is a slice of structs.
	// It is treated as invalid to avoid being mistakenly passed when []PropertyList was intended.
	GetMulti(ctx context.Context, keys []Key, dst interface{}) error

	// Put saves the entity src into the datastore with key k.
	// src must be a struct pointer or implement PropertyLoadSaver; if a struct pointer then any unexported fields of that struct will be skipped.
	// If k is an incomplete key, the returned key will be a unique key generated by the datastore.
	Put(ctx context.Context, key Key, src interface{}) (Key, error)

	// PutMulti is a batch version of Put.
	//
	// src must satisfy the same conditions as the dst argument to GetMulti.
	PutMulti(ctx context.Context, keys []Key, src interface{}) ([]Key, error)

	// Delete deletes the entity for the given key.
	Delete(ctx context.Context, key Key) error

	// DeleteMulti is a batch version of Delete.
	DeleteMulti(ctx context.Context, keys []Key) error

	// NewTransaction starts a new transaction.
	NewTransaction(ctx context.Context) (Transaction, error)

	// RunInTransaction runs f in a transaction. f is invoked with a Transaction that f should use for all the transaction's datastore operations.
	//
	// f must not call Commit or Rollback on the provided Transaction.
	//
	// If f returns nil, RunInTransaction commits the transaction, returning the Commit and a nil error if it succeeds.
	// If the commit fails due to a conflicting transaction, RunInTransaction gives up and returns ErrConcurrentTransaction immediately.
	// If you want to retry operation, You have to retry by yourself.
	//
	// If f returns non-nil, then the transaction will be rolled back and RunInTransaction will return the same error.
	//
	// Note that when f returns, the transaction is not committed. Calling code must not assume that any of f's changes have been committed until RunInTransaction returns nil.
	RunInTransaction(ctx context.Context, f func(tx Transaction) error) (Commit, error)

	// Run runs the given query in the given context.
	Run(ctx context.Context, q Query) Iterator

	// AllocateIDs accepts a slice of incomplete keys and returns a slice of complete keys that are guaranteed to be valid in the datastore.
	AllocateIDs(ctx context.Context, keys []Key) ([]Key, error)

	// Count returns the number of results for the given query.
	//
	// The running time and number of API calls made by Count scale linearly with with the sum of the query's offset and limit.
	// Unless the result count is expected to be small, it is best to specify a limit; otherwise Count will continue until it finishes counting or the provided context expires.
	Count(ctx context.Context, q Query) (int, error)

	// GetAll runs the provided query in the given context and returns all keys that match that query, as well as appending the values to dst.
	//
	// dst must have type *[]S or *[]*S or *[]P, for some struct type S or some non-interface, non-pointer type P such that P or *P implements PropertyLoadSaver.
	//
	// As a special case, *PropertyList is an invalid type for dst, even though a PropertyList is a slice of structs.
	// It is treated as invalid to avoid being mistakenly passed when *[]PropertyList was intended.
	//
	// The keys returned by GetAll will be in a 1-1 correspondence with the entities added to dst.
	//
	// If q is a “keys-only” query, GetAll ignores dst and only returns the keys.
	//
	// The running time and number of API calls made by GetAll scale linearly with with the sum of the query's offset and limit.
	// Unless the result count is expected to be small, it is best to specify a limit; otherwise GetAll will continue until it finishes collecting results or the provided context expires.
	GetAll(ctx context.Context, q Query, dst interface{}) ([]Key, error)

	// IncompleteKey creates a new incomplete key.
	// The supplied kind cannot be empty.
	// The namespace of the new key is empty.
	IncompleteKey(kind string, parent Key) Key

	// NameKey creates a new key with a name.
	// The supplied kind cannot be empty.
	// The supplied parent must either be a complete key or nil.
	// The namespace of the new key is empty.
	NameKey(kind, name string, parent Key) Key

	// IDKey creates a new key with an ID.
	// The supplied kind cannot be empty.
	// The supplied parent must either be a complete key or nil.
	// The namespace of the new key is empty.
	IDKey(kind string, id int64, parent Key) Key

	// NewQuery creates a new Query for a specific entity kind.
	//
	// An empty kind means to return all entities, including entities created and managed by other App Engine features, and is called a kindless query.
	// Kindless queries cannot include filters or sort orders on property values.
	NewQuery(kind string) Query

	// Close closes the Client.
	Close() error

	// DecodeKey decodes a key from the opaque representation returned by Encode.
	DecodeKey(encoded string) (Key, error)

	// DecodeCursor from its base-64 string representation.
	DecodeCursor(s string) (Cursor, error)

	// Batch creates batch mode objects.
	Batch() *Batch
	// AppendMiddleware to client.
	// Middleware will apply First-In First-Apply
	AppendMiddleware(middleware Middleware)
	// RemoveMiddleware from client.
	RemoveMiddleware(middleware Middleware) bool
	// Context returns this client's context.
	Context() context.Context
	// SetContext to this client.
	SetContext(ctx context.Context)
}

Client is a client for reading and writing data in a datastore dataset.

type ClientGenerator

type ClientGenerator func(ctx context.Context, opts ...ClientOption) (Client, error)

ClientGenerator represents the type of function for generating Client.

Deprecated:

var FromContext ClientGenerator

FromContext provides default ClientGenerator. this variable will be injected by go.mercari.io/datastore/aedatastore or go.mercari.io/datastore/clouddatastore package's init function.

Deprecated: use aedatastore.FromContext or clouddatastore.FromContext instead.

type ClientOption

type ClientOption interface {
	Apply(*internal.ClientSettings)
}

A ClientOption is an option for a Datastore client.

func WithCredentialsFile

func WithCredentialsFile(filename string) ClientOption

WithCredentialsFile returns a ClientOption that authenticates API calls with the given service account or refresh token JSON credentials file.

func WithGRPCDialOption

func WithGRPCDialOption(opt grpc.DialOption) ClientOption

WithGRPCDialOption returns a ClientOption that appends a new grpc.DialOption to an underlying gRPC dial. It does not work with WithGRPCConn.

func WithHTTPClient

func WithHTTPClient(client *http.Client) ClientOption

WithHTTPClient returns a ClientOption that specifies the HTTP client to use as the basis of communications. This option may only be used with services that support HTTP as their communication transport. When used, the WithHTTPClient option takes precedent over all other supplied options.

func WithProjectID

func WithProjectID(projectID string) ClientOption

WithProjectID returns a ClientOption that specifies ProjectID to be used in client.

func WithScopes

func WithScopes(scope ...string) ClientOption

WithScopes returns a ClientOption that overrides the default OAuth2 scopes to be used for a service.

func WithTokenSource

func WithTokenSource(s oauth2.TokenSource) ClientOption

WithTokenSource returns a ClientOption that specifies an OAuth2 token source to be used as the basis for authentication.

type Commit

type Commit interface {
	Key(p PendingKey) Key
}

Commit represents the result of a committed transaction.

type Cursor

type Cursor interface {
	String() string
}

Cursor is an iterator's position. It can be converted to and from an opaque string. A cursor can be used from different HTTP requests, but only with a query with the same kind, ancestor, filter and order constraints.

The zero Cursor can be used to indicate that there is no start and/or end constraint for a query.

type Entity

type Entity struct {
	Key        Key
	Properties []Property
}

An Entity is the value type for a nested struct. This type is only used for a Property's Value.

type ErrFieldMismatch

type ErrFieldMismatch struct {
	StructType reflect.Type
	FieldName  string
	Reason     string
}

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. StructType is the type of the struct pointed to by the destination argument passed to Get or to Iterator.Next.

func (*ErrFieldMismatch) Error

func (e *ErrFieldMismatch) Error() string

type GeoPoint

type GeoPoint struct {
	Lat, Lng float64
}

GeoPoint represents a location as latitude/longitude in degrees.

type Iterator

type Iterator interface {
	// Next returns the key of the next result. When there are no more results,
	// iterator.Done is returned as the error.
	//
	// If the query is not keys only and dst is non-nil, it also loads the entity
	// stored for that key into the struct pointer or PropertyLoadSaver dst, with
	// the same semantics and possible errors as for the Get function.
	Next(dst interface{}) (Key, error)
	// Cursor returns a cursor for the iterator's current location.
	Cursor() (Cursor, error)
}

Iterator is the result of running a query.

type Key

type Key interface {
	Kind() string
	ID() int64
	Name() string
	ParentKey() Key
	Namespace() string
	SetNamespace(namespace string)

	String() string
	GobEncode() ([]byte, error)
	GobDecode(buf []byte) error
	MarshalJSON() ([]byte, error)
	UnmarshalJSON(buf []byte) error
	Encode() string
	Equal(o Key) bool
	Incomplete() bool
}

Key represents the datastore key for a stored entity.

type KeyLoader

type KeyLoader interface {
	// PropertyLoadSaver is embedded because a KeyLoader
	// must also always implement PropertyLoadSaver.
	PropertyLoadSaver
	LoadKey(ctx context.Context, k Key) error
}

KeyLoader can store a Key.

type Middleware

type Middleware interface {
	// AllocateIDs intercepts AllocateIDs operation.
	AllocateIDs(info *MiddlewareInfo, keys []Key) ([]Key, error)
	// PutMultiWithoutTx intercepts PutMulti without Transaction operation.
	PutMultiWithoutTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) ([]Key, error)
	// PutMultiWithTx intercepts PutMulti with Transaction operation.
	PutMultiWithTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) ([]PendingKey, error)
	// GetMultiWithoutTx intercepts GetMulti without Transaction operation.
	GetMultiWithoutTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) error
	// GetMultiWithTx intercepts GetMulti with Transaction operation.
	GetMultiWithTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) error
	// DeleteMultiWithoutTx intercepts DeleteMulti without Transaction operation.
	DeleteMultiWithoutTx(info *MiddlewareInfo, keys []Key) error
	// DeleteMultiWithTx intercepts DeleteMulti with Transaction operation.
	DeleteMultiWithTx(info *MiddlewareInfo, keys []Key) error
	// PostCommit will kicked after Transaction commit.
	PostCommit(info *MiddlewareInfo, tx Transaction, commit Commit) error
	// PostRollback will kicked after Transaction rollback.
	PostRollback(info *MiddlewareInfo, tx Transaction) error
	// Run intercepts Run query operation.
	Run(info *MiddlewareInfo, q Query, qDump *QueryDump) Iterator
	// GetAll intercepts GetAll operation.
	GetAll(info *MiddlewareInfo, q Query, qDump *QueryDump, psList *[]PropertyList) ([]Key, error)
	// Next intercepts Next operation.
	Next(info *MiddlewareInfo, q Query, qDump *QueryDump, iter Iterator, ps *PropertyList) (Key, error)
	// Count intercepts Count operation.
	Count(info *MiddlewareInfo, q Query, qDump *QueryDump) (int, error)
}

Middleware hooks to the Datastore's RPC and It can modify arguments and return values. see https://godoc.org/go.mercari.io/datastore/dsmiddleware

type MiddlewareInfo

type MiddlewareInfo struct {
	Context     context.Context
	Client      Client
	Transaction Transaction
	Next        Middleware
}

MiddlewareInfo provides RPC's processing state.

type MultiError

type MultiError []error

MultiError is returned by batch operations when there are errors with particular elements. Errors will be in a one-to-one correspondence with the input elements; successful elements will have a nil entry.

func (MultiError) Error

func (m MultiError) Error() string

type PendingKey

type PendingKey interface {
	StoredContext() context.Context
}

PendingKey represents the key for newly-inserted entity. It can be resolved into a Key by calling the Key method of Commit.

type Property

type Property struct {
	// Name is the property name.
	Name string
	// Value is the property value. The valid types are:
	//	- int64
	//	- bool
	//	- string
	//	- float64
	//	- Key
	//	- time.Time (retrieved as local time)
	//	- GeoPoint
	//	- []byte (up to 1 megabyte in length)
	//	- *Entity (representing a nested struct)
	// Value can also be:
	//	- []interface{} where each element is one of the above types
	// This set is smaller than the set of valid struct field types that the
	// datastore can load and save. A Value's type must be explicitly on
	// the list above; it is not sufficient for the underlying type to be
	// on that list. For example, a Value of "type myInt64 int64" is
	// invalid. Smaller-width integers and floats are also invalid. Again,
	// this is more restrictive than the set of valid struct field types.
	//
	// A Value will have an opaque type when loading entities from an index,
	// such as via a projection query. Load entities into a struct instead
	// of a PropertyLoadSaver when using a projection query.
	//
	// A Value may also be the nil interface value; this is equivalent to
	// Python's None but not directly representable by a Go struct. Loading
	// a nil-valued property into a struct will set that field to the zero
	// value.
	Value interface{}
	// NoIndex is whether the datastore cannot index this property.
	// If NoIndex is set to false, []byte and string values are limited to
	// 1500 bytes.
	NoIndex bool
}

Property is a name/value pair plus some metadata. A datastore entity's contents are loaded and saved as a sequence of Properties. Each property name must be unique within an entity.

func SaveStruct

func SaveStruct(ctx context.Context, src interface{}) ([]Property, error)

SaveStruct returns the properties from src as a slice of Properties. src must be a struct pointer.

type PropertyList

type PropertyList []Property

PropertyList converts a []Property to implement PropertyLoadSaver.

func (*PropertyList) Load

func (l *PropertyList) Load(ctx context.Context, p []Property) error

Load loads all of the provided properties into l. It does not first reset *l to an empty slice.

func (*PropertyList) Save

func (l *PropertyList) Save(ctx context.Context) ([]Property, error)

Save saves all of l's properties as a slice of Properties.

type PropertyLoadSaver

type PropertyLoadSaver interface {
	Load(ctx context.Context, ps []Property) error
	Save(ctx context.Context) ([]Property, error)
}

PropertyLoadSaver can be converted from and to a slice of Properties.

type PropertyTranslator

type PropertyTranslator interface {
	ToPropertyValue(ctx context.Context) (interface{}, error)
	FromPropertyValue(ctx context.Context, p Property) (dst interface{}, err error)
}

PropertyTranslator is for converting the value of Property when saving and loading.

Example

package main

import (
	"context"
	"fmt"

	"go.mercari.io/datastore"
	"go.mercari.io/datastore/clouddatastore"
	"go.mercari.io/datastore/internal/testutils"
)

type UserID int64
type UserIDs []UserID

type contextClient struct{}

func (id UserID) ToPropertyValue(ctx context.Context) (interface{}, error) {

	client := ctx.Value(contextClient{}).(datastore.Client)
	key := client.IDKey("User", int64(id), nil)
	return key, nil
}

func (id UserID) FromPropertyValue(ctx context.Context, p datastore.Property) (dst interface{}, err error) {

	key, ok := p.Value.(datastore.Key)
	if !ok {
		return nil, datastore.ErrInvalidEntityType
	}
	return UserID(key.ID()), nil
}

func (ids UserIDs) ToPropertyValue(ctx context.Context) (interface{}, error) {

	client := ctx.Value(contextClient{}).(datastore.Client)
	keys := make([]datastore.Key, 0, len(ids))
	for _, id := range ids {
		keys = append(keys, client.IDKey("User", int64(id), nil))
	}
	return keys, nil
}

func (ids UserIDs) FromPropertyValue(ctx context.Context, p datastore.Property) (dst interface{}, err error) {

	keys, ok := p.Value.([]datastore.Key)
	if !ok {
		return nil, datastore.ErrInvalidEntityType
	}
	newIDs := make(UserIDs, 0, len(keys))
	for _, key := range keys {
		newIDs = append(ids, UserID(key.ID()))
	}
	return newIDs, nil
}

func main() {
	ctx := context.Background()
	client, err := clouddatastore.FromContext(ctx)
	if err != nil {
		panic(err)
	}
	defer client.Close()
	defer testutils.CleanUpAllEntities(ctx, client)

	ctx = context.WithValue(ctx, contextClient{}, client)

	// Each fields are saved as datastore.Key and [] datastore.Key on Datastore.
	type Group struct {
		OwnerID   UserID
		MemberIDs UserIDs
	}

	entity, err := datastore.SaveEntity(
		ctx, client.IncompleteKey("Group", nil),
		&Group{
			OwnerID:   147,
			MemberIDs: UserIDs{147, 258, 369},
		},
	)
	if err != nil {
		panic(err)
	}

	if key, ok := entity.Properties[0].Value.(datastore.Key); !ok {
		panic("unexpected state")
	} else {
		fmt.Println("OwnerID", key.ID())
	}
	if keys, ok := entity.Properties[1].Value.([]datastore.Key); !ok {
		panic("unexpected state")
	} else {
		for _, key := range keys {
			fmt.Println("MemberID", key.ID())
		}
	}

}

Output:

OwnerID 147
MemberID 147
MemberID 258
MemberID 369

type Query

type Query interface {
	Ancestor(ancestor Key) Query
	EventualConsistency() Query
	Namespace(ns string) Query
	Transaction(t Transaction) Query
	Filter(filterStr string, value interface{}) Query
	Order(fieldName string) Query
	Project(fieldNames ...string) Query
	Distinct() Query
	DistinctOn(fieldNames ...string) Query
	KeysOnly() Query
	Limit(limit int) Query
	Offset(offset int) Query
	Start(c Cursor) Query
	End(c Cursor) Query

	Dump() *QueryDump
}

Query represents a datastore query.

type QueryDump

type QueryDump struct {
	Kind                string
	Ancestor            Key
	EventualConsistency bool
	Namespace           string
	Transaction         Transaction
	Filter              []*QueryFilterCondition
	Order               []string
	Project             []string
	DistinctOn          []string
	Distinct            bool
	KeysOnly            bool
	Limit               int
	Offset              int
	Start               Cursor
	End                 Cursor
}

QueryDump provides information of executed query.

func (*QueryDump) String

func (dump *QueryDump) String() string

type QueryFilterCondition

type QueryFilterCondition struct {
	Filter string
	Value  interface{}
}

QueryFilterCondition provides information of filter of query.

type Transaction

type Transaction interface {
	// Get is the transaction-specific version of the package function Get.
	// All reads performed during the transaction will come from a single consistent snapshot.
	// Furthermore, if the transaction is set to a serializable isolation level,
	// another transaction cannot concurrently modify the data that is read or modified by this transaction.
	Get(key Key, dst interface{}) error
	// GetMulti is a batch version of Get.
	GetMulti(keys []Key, dst interface{}) error
	// Put is the transaction-specific version of the package function Put.
	//
	// Put returns a PendingKey which can be resolved into a Key using the return value from a successful Commit.
	// If key is an incomplete key, the returned pending key will resolve to a unique key generated by the datastore.
	Put(key Key, src interface{}) (PendingKey, error)
	// PutMulti is a batch version of Put. One PendingKey is returned for each element of src in the same order.
	PutMulti(keys []Key, src interface{}) ([]PendingKey, error)
	// Delete is the transaction-specific version of the package function Delete.
	// Delete enqueues the deletion of the entity for the given key,
	// to be committed atomically upon calling Commit.
	Delete(key Key) error
	// DeleteMulti is a batch version of Delete.
	DeleteMulti(keys []Key) error

	// Commit applies the enqueued operations atomically.
	Commit() (Commit, error)
	// Rollback abandons a pending transaction.
	Rollback() error

	Batch() *TransactionBatch
}

Transaction represents a set of datastore operations to be committed atomically.

Operations are enqueued by calling the Put and Delete methods on Transaction (or their Multi-equivalents). These operations are only committed when the Commit method is invoked. To ensure consistency, reads must be performed by using Transaction's Get method or by using the Transaction method when building a query.

A Transaction must be committed or rolled back exactly once.

type TransactionBatch

type TransactionBatch struct {
	Transaction Transaction
	// contains filtered or unexported fields
}

TransactionBatch provides Batch operation under Transaction. TransactionBatch does nothing until you call Exec(). This helps to reduce the number of RPCs.

func (*TransactionBatch) Delete

func (b *TransactionBatch) Delete(key Key, h BatchErrHandler)

Delete Entity operation into the queue.

func (*TransactionBatch) Exec

func (b *TransactionBatch) Exec() error

Exec will perform all the processing that was queued. This process is done recursively until the queue is empty. The return value may be MultiError, but the order of contents is not guaranteed. Exec() doesn't call Commit() or Rollback(), You should call that manually.

func (*TransactionBatch) Get

func (b *TransactionBatch) Get(key Key, dst interface{}, h BatchErrHandler)

Get Entity operation into the queue.

func (*TransactionBatch) Put

func (b *TransactionBatch) Put(key Key, src interface{}, h TxBatchPutHandler)

Put Entity operation into the queue. This operation doesn't Put to Datastore immediately. If a h is provided, it passes the processing result to the handler, and treats the return value as the value of the result of Putting.

type TxBatchPutHandler

type TxBatchPutHandler func(pKey PendingKey, err error) error

TxBatchPutHandler represents Entity's individual callback when batching Put with transaction processing.