datastore

package module
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 24, 2018 License: MIT Imports: 16 Imported by: 0

README

Datastore Wrapper Go Documentation CircleCI

🚧 This package is unstable 🚧 github repo

(AppEngine | Cloud) Datastore wrapper for Go 👉

Simple. Happy. Respect standard library.

$ go get -u go.mercari.io/datastore

Feature

DO
  • Wrap google.golang.org/appengine/datastore and cloud.google.com/go/datastore
    • keep key behavior
    • align to cloud.google.com/go/datastore first
  • Re-implement datastore package
  • Re-implement datastore.SaveStruct & LoadStruct
    • Ignore unmapped property
    • Add PropertyTranslator interface
      • Convert types like mytime.Unix to time.Time and reverse it
      • Rename property like CreatedAt to createdAt or created_at and reverse it
  • Re-implement PropertyLoadSaver
    • Pass context.Context to Save & Load method
  • Add retry feature to each RPC
    • e.g. Retry AllocateID when it failed
  • Add middleware layer
    • About...
      • Local Cache
      • AE Memcache
      • Logging
      • Retry
      • etc...
    • Easy to ON/OFF switching
  • Add some useful methods
    • aedatastore/TransactionContext
DON'T
  • have utility functions
  • support firestore

Restriction

  • aedatastore package
    • When using slice of struct, MUST specified datastore:",flatten" option.
      • original (ae & cloud) datastore.SaveStruct have different behaviors.
      • see aeprodtest/main.go /api/test3

How To Use

TODO

Best Practice

TODO

Project Navigation

  • aedatastore
    • Datastore Wrapper implementation for AppEngine Datastore
  • aeprodtest
    • Test code that works in AppEngine production environment
  • boom
    • goon likes interface for this package
  • clouddatastore
    • Datastore Wrapper implementation for Cloud Datastore
  • dsmiddleware
    • Various processing can be intercepted
    • dsmiddleware/aememcache
    • dsmiddleware/chaosrpc
      • Randomly generate an RPC error!
    • dsmiddleware/dslog
      • Datastore operation logging in cache layer
    • dsmiddleware/fishbone
      • Replace query by KeysOnly query with Get ops
    • dsmiddleware/localcache
      • Entity caching with machine local memory
    • dsmiddleware/noop
      • noop middleware 😉
    • dsmiddleware/rediscache
      • Entity caching with Redis
    • dsmiddleware/rpcretry
      • Retry RPC when an error occurred
    • dsmiddleware/storagecache
      • Entity cacheing base code
  • internal
    • internal package
  • testbed
    • Testbed to check the behavior of AE Datastore and Cloud Datastore
  • testsuite

TODO

  • Write more documents

Committers

Contribution

Please read the CLA below carefully before submitting your contribution.

https://www.mercari.com/cla/

Setup environment & Run tests
  • requirements
    • gcloud sdk
      • gcloud components install app-engine-go
      • gcloud components install beta cloud-datastore-emulator
    • dep
      • go get -u github.com/golang/dep/cmd/dep
  1. Testing in local
$ ./setup.sh # exec once
$ ./serve.sh # exec in background
$ ./test.sh
  1. Testing with Circle CI CLI
$ circleci build

License

Copyright 2017 Mercari, Inc.

Licensed under the MIT License.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidEntityType = errors.New("datastore: invalid entity type")
	ErrInvalidKey        = errors.New("datastore: invalid key")
	ErrNoSuchEntity      = errors.New("datastore: no such entity")
)
View Source 
var ErrConcurrentTransaction = errors.New("datastore: concurrent transaction")
View Source 
var LoadEntity = loadEntity

TODO hide LoadEntity project outside

View Source 
var SaveEntity = saveEntity
View Source 
var SuppressErrFieldMismatch = true

Functions

func LoadStruct 

func LoadStruct(ctx context.Context, dst interface{}, p []Property) error

LoadStruct loads the properties from p to dst. dst must be a struct pointer.

The values of dst's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each LoadStruct call.

Types

type Batch 

type Batch struct {
	Client Client
	// contains filtered or unexported fields
}

func (*Batch) Delete 

func (b *Batch) Delete(key Key, h BatchErrHandler)

func (*Batch) Exec 

func (b *Batch) Exec(ctx context.Context) error

func (*Batch) Get 

func (b *Batch) Get(key Key, dst interface{}, h BatchErrHandler)

func (*Batch) Put 

func (b *Batch) Put(key Key, src interface{}, h BatchPutHandler)

type BatchErrHandler added in v0.9.0 

type BatchErrHandler func(err error) error

type BatchPutHandler added in v0.9.0 

type BatchPutHandler func(key Key, err error) error

type Client 

type Client interface {
	Get(ctx context.Context, key Key, dst interface{}) error
	GetMulti(ctx context.Context, keys []Key, dst interface{}) error
	Put(ctx context.Context, key Key, src interface{}) (Key, error)
	PutMulti(ctx context.Context, keys []Key, src interface{}) ([]Key, error)
	Delete(ctx context.Context, key Key) error
	DeleteMulti(ctx context.Context, keys []Key) error

	NewTransaction(ctx context.Context) (Transaction, error)
	RunInTransaction(ctx context.Context, f func(tx Transaction) error) (Commit, error)
	Run(ctx context.Context, q Query) Iterator
	AllocatedIDs(ctx context.Context, keys []Key) ([]Key, error)
	Count(ctx context.Context, q Query) (int, error)
	GetAll(ctx context.Context, q Query, dst interface{}) ([]Key, error)

	IncompleteKey(kind string, parent Key) Key
	NameKey(kind, name string, parent Key) Key
	IDKey(kind string, id int64, parent Key) Key

	NewQuery(kind string) Query

	Close() error

	DecodeKey(encoded string) (Key, error)
	DecodeCursor(s string) (Cursor, error)

	Batch() *Batch
	AppendMiddleware(middleware Middleware) // NOTE First-In First-Apply
	RemoveMiddleware(middleware Middleware) bool
	Context() context.Context
	SetContext(ctx context.Context)
}

type ClientGenerator 

type ClientGenerator func(ctx context.Context, opts ...ClientOption) (Client, error)
var FromContext ClientGenerator

type ClientOption 

type ClientOption interface {
	Apply(*internal.ClientSettings)
}

func WithCredentialsFile 

func WithCredentialsFile(filename string) ClientOption

WithCredentialsFile returns a ClientOption that authenticates API calls with the given service account or refresh token JSON credentials file.

func WithHTTPClient 

func WithHTTPClient(client *http.Client) ClientOption

WithHTTPClient returns a ClientOption that specifies the HTTP client to use as the basis of communications. This option may only be used with services that support HTTP as their communication transport. When used, the WithHTTPClient option takes precedent over all other supplied options.

func WithProjectID 

func WithProjectID(projectID string) ClientOption

func WithScopes 

func WithScopes(scope ...string) ClientOption

WithScopes returns a ClientOption that overrides the default OAuth2 scopes to be used for a service.

func WithTokenSource 

func WithTokenSource(s oauth2.TokenSource) ClientOption

WithTokenSource returns a ClientOption that specifies an OAuth2 token source to be used as the basis for authentication.

type Commit 

type Commit interface {
	Key(p PendingKey) Key
}

type Cursor 

type Cursor interface {
	String() string
}

type Entity 

type Entity struct {
	Key        Key
	Properties []Property
}

type ErrFieldMismatch 

type ErrFieldMismatch struct {
	StructType reflect.Type
	FieldName  string
	Reason     string
}

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. StructType is the type of the struct pointed to by the destination argument passed to Get or to Iterator.Next.

func (*ErrFieldMismatch) Error 

func (e *ErrFieldMismatch) Error() string

type GeoPoint 

type GeoPoint struct {
	Lat, Lng float64
}

type Iterator 

type Iterator interface {
	Next(dst interface{}) (Key, error)
	Cursor() (Cursor, error)
}

type Key 

type Key interface {
	Kind() string
	ID() int64
	Name() string
	ParentKey() Key
	Namespace() string
	SetNamespace(namespace string)

	String() string
	GobEncode() ([]byte, error)
	GobDecode(buf []byte) error
	MarshalJSON() ([]byte, error)
	UnmarshalJSON(buf []byte) error
	Encode() string
	Equal(o Key) bool
	Incomplete() bool
}

type KeyLoader 

type KeyLoader interface {
	PropertyLoadSaver
	LoadKey(ctx context.Context, k Key) error
}

type Middleware added in v0.11.0 

type Middleware interface {
	AllocateIDs(info *MiddlewareInfo, keys []Key) ([]Key, error)
	PutMultiWithoutTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) ([]Key, error)
	PutMultiWithTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) ([]PendingKey, error)
	GetMultiWithoutTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) error
	GetMultiWithTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) error
	DeleteMultiWithoutTx(info *MiddlewareInfo, keys []Key) error
	DeleteMultiWithTx(info *MiddlewareInfo, keys []Key) error
	PostCommit(info *MiddlewareInfo, tx Transaction, commit Commit) error
	PostRollback(info *MiddlewareInfo, tx Transaction) error
	Run(info *MiddlewareInfo, q Query, qDump *QueryDump) Iterator
	GetAll(info *MiddlewareInfo, q Query, qDump *QueryDump, psList *[]PropertyList) ([]Key, error)
	Next(info *MiddlewareInfo, q Query, qDump *QueryDump, iter Iterator, ps *PropertyList) (Key, error)
	Count(info *MiddlewareInfo, q Query, qDump *QueryDump) (int, error)
}

type MiddlewareInfo added in v0.11.0 

type MiddlewareInfo struct {
	Context     context.Context
	Client      Client
	Transaction Transaction
	Next        Middleware
}

type MultiError 

type MultiError []error

MultiError is returned by batch operations when there are errors with particular elements. Errors will be in a one-to-one correspondence with the input elements; successful elements will have a nil entry.

func (MultiError) Error 

func (m MultiError) Error() string

type PendingKey 

type PendingKey interface {
	StoredContext() context.Context
}

type Property 

type Property struct {
	Name    string
	Value   interface{}
	NoIndex bool
}

func SaveStruct 

func SaveStruct(ctx context.Context, src interface{}) ([]Property, error)

SaveStruct returns the properties from src as a slice of Properties. src must be a struct pointer.

type PropertyList 

type PropertyList []Property

func (*PropertyList) Load 

func (l *PropertyList) Load(ctx context.Context, p []Property) error

Load loads all of the provided properties into l. It does not first reset *l to an empty slice.

func (*PropertyList) Save 

func (l *PropertyList) Save(ctx context.Context) ([]Property, error)

Save saves all of l's properties as a slice of Properties.

type PropertyLoadSaver 

type PropertyLoadSaver interface {
	Load(ctx context.Context, ps []Property) error
	Save(ctx context.Context) ([]Property, error)
}

type PropertyTranslator 

type PropertyTranslator interface {
	ToPropertyValue(ctx context.Context) (interface{}, error)
	FromPropertyValue(ctx context.Context, p Property) (dst interface{}, err error)
}

type Query 

type Query interface {
	Ancestor(ancestor Key) Query
	EventualConsistency() Query
	Namespace(ns string) Query
	Transaction(t Transaction) Query
	Filter(filterStr string, value interface{}) Query
	Order(fieldName string) Query
	Project(fieldNames ...string) Query
	Distinct() Query
	// NOT IMPLEMENTED ON APPENGINE DistinctOn(fieldNames ...string) *Query
	KeysOnly() Query
	Limit(limit int) Query
	Offset(offset int) Query
	Start(c Cursor) Query
	End(c Cursor) Query

	Dump() *QueryDump
}

type QueryDump added in v0.7.0 

type QueryDump struct {
	Kind                string
	Ancestor            Key
	EventualConsistency bool
	Namespace           string
	Transaction         Transaction
	Filter              []*QueryFilterCondition
	Order               []string
	Project             []string
	Distinct            bool
	KeysOnly            bool
	Limit               int
	Offset              int
	Start               Cursor
	End                 Cursor
}

func (*QueryDump) String added in v0.7.0 

func (dump *QueryDump) String() string

type QueryFilterCondition added in v0.7.0 

type QueryFilterCondition struct {
	Filter string
	Value  interface{}
}

type Transaction 

type Transaction interface {
	Get(key Key, dst interface{}) error
	GetMulti(keys []Key, dst interface{}) error
	Put(key Key, src interface{}) (PendingKey, error)
	PutMulti(keys []Key, src interface{}) ([]PendingKey, error)
	Delete(key Key) error
	DeleteMulti(keys []Key) error

	Commit() (Commit, error)
	Rollback() error

	Batch() *TransactionBatch
}

type TransactionBatch 

type TransactionBatch struct {
	Transaction Transaction
	// contains filtered or unexported fields
}

func (*TransactionBatch) Delete 

func (b *TransactionBatch) Delete(key Key, h BatchErrHandler)

func (*TransactionBatch) Exec 

func (b *TransactionBatch) Exec() error

func (*TransactionBatch) Get 

func (b *TransactionBatch) Get(key Key, dst interface{}, h BatchErrHandler)

func (*TransactionBatch) Put 

func (b *TransactionBatch) Put(key Key, src interface{}, h TxBatchPutHandler)

type TxBatchPutHandler added in v0.9.0 

type TxBatchPutHandler func(pKey PendingKey, err error) error

Source Files

View all Source files

Directories

Path Synopsis
dsmiddleware
aememcache
chaosrpc
dslog
fishbone
localcache
noop
rediscache
rpcretry
storagecache
c/atomiccache
Package atomiccache provides a map-based cache that supports very fast reads.
 Package atomiccache provides a map-based cache that supports very fast reads.
c/fields
Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
 Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
pb/memcache
Package memcache is a generated protocol buffer package.
 Package memcache is a generated protocol buffer package.
shared
testutils
dsmiddleware/dslog
dsmiddleware/fishbone
dsmiddleware/localcache
dsmiddleware/rpcretry
favcliptools
realworld/recursive_batch
realworld/tbf

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.