datastore

package module
v0.0.0-...-c70f986 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 19, 2023 License: BSD-3-Clause Imports: 15 Imported by: 7

Documentation

Overview

Package datastore offers common datastore API with multiple store implementations:

  • CloudStore is a Google datastore implementation.
  • FileStore is a simple file-based store implementation.

Index

Constants

View Source
const (
	MaxKeys = 500     // Maximum number of datastore keys to retrieve in one go.
	MaxBlob = 1000000 // 1MB
)

CloudStore limits.

View Source
const (
	EpochStart = 1483228800    // Start of the AusOcean epoch, namely 2017-01-01 00:00:00+Z.
	EpochEnd   = math.MaxInt64 // End of the epoch.

)

FileStore constants.

Variables

View Source
var (
	ErrDecoding        = errors.New("decoding error")
	ErrUnimplemented   = errors.New("unimplemented feature")
	ErrInvalidField    = errors.New("invalid field")
	ErrInvalidFilter   = errors.New("invalid filter")
	ErrInvalidOperator = errors.New("invalid operator")
	ErrInvalidValue    = errors.New("invalid value")
	ErrOperatorMissing = errors.New("operator missing")
	ErrNoSuchEntity    = errors.New("no such entity")
	ErrEntityExists    = errors.New("entity exists")
)

Functions

func DeleteMulti

func DeleteMulti(ctx context.Context, store Store, keys []*Key) (int, error)

DeleteMulti is a wrapper for store.DeleteMulti which returns the number of deletions.

func IDKey

func IDKey(id, ts, st int64) int64

IDKey makes a datastore ID key by combining an ID, a Unix timestamp and an optional subtime (st). The latter is used to disambiguate otherwise-identical timestamps.

  • The least significant 32 bits of the ID.
  • The timestamp from the start of the "AusOcean epoch", 2017-01-01 00:00:00+Z (29 bits).
  • The subtime (3 bits).

func RegisterEntity

func RegisterEntity(kind string, construct func() Entity)

RegisterEntity registers a new kind of entity and its constructor.

func SplitIDKey

func SplitIDKey(id int64) (int64, int64, int64)

SplitIDKey is the inverse of IDKey and splits an ID key into its parts. See IDKey.

Types

type CloudQuery

type CloudQuery struct {
	// contains filtered or unexported fields
}

CloudQuery implements Query for the Google Cloud Datastore.

func (*CloudQuery) Filter

func (q *CloudQuery) Filter(filterStr string, value interface{}) error

func (*CloudQuery) Limit

func (q *CloudQuery) Limit(limit int)

func (*CloudQuery) Order

func (q *CloudQuery) Order(fieldName string)

type CloudStore

type CloudStore struct {
	// contains filtered or unexported fields
}

CloudStore implements Store for the Google Cloud Datastore.

func (*CloudStore) Create

func (s *CloudStore) Create(ctx context.Context, key *Key, src Entity) error

func (*CloudStore) DeleteMulti

func (s *CloudStore) DeleteMulti(ctx context.Context, keys []*Key) error

func (*CloudStore) Get

func (s *CloudStore) Get(ctx context.Context, key *Key, dst Entity) error

func (*CloudStore) GetAll

func (s *CloudStore) GetAll(ctx context.Context, query Query, dst interface{}) ([]*Key, error)

func (*CloudStore) IDKey

func (s *CloudStore) IDKey(kind string, id int64) *Key

IDKey returns an ID key given a kind and an int64 ID.

func (*CloudStore) NameKey

func (s *CloudStore) NameKey(kind, name string) *Key

NameKey returns an name key given a kind and a (string) name.

func (*CloudStore) NewQuery

func (s *CloudStore) NewQuery(kind string, keysOnly bool, keyParts ...string) Query

NewQuery returns a new CloudQuery and is a wrapper for datastore.NewQuery. If keysOnly is true the query is set to keys only, but keyParts are ignored.

func (*CloudStore) Put

func (s *CloudStore) Put(ctx context.Context, key *Key, src Entity) (*Key, error)

func (*CloudStore) Update

func (s *CloudStore) Update(ctx context.Context, key *Key, fn func(Entity), dst Entity) error

type Entity

type Entity interface {
	Encode() []byte      // Encode entity into bytes.
	Decode([]byte) error // Decode bytes into entity.
}

Entity defines the common interface for our datastore entities.

type FileQuery

type FileQuery struct {
	// contains filtered or unexported fields
}

FileQuery implements Query for FileStore.

func (*FileQuery) Filter

func (q *FileQuery) Filter(filterStr string, value interface{}) error

Filter implements FileQuery matching against key parts. This function transforms certain properties commonly used in queries:

  • Properties named ID or MID are reduced to their least-significant 32 bits.
  • Properties named Timestamp are converted from the Unix epoch to the AusOcean epoch.

func (*FileQuery) Limit

func (q *FileQuery) Limit(limit int)

NOTE: Limit has not been implemented for file stores.

func (*FileQuery) Order

func (q *FileQuery) Order(fieldName string)

type FileStore

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore implements Store for file storage. Each entity is represented as a file named <key> under the directory <dir>/<id>/<kind>, where <kind> and <key> are the entity kind and key respectively.

FileStore implements a simple form of indexing based on the period-separated parts of key names. For example, a User has the key structure <Skey>.<Email>. Filestore queries that utilize indexes must specify the optional key parts when constructing the query.

q = store.NewQuery(ctx, "User", "Skey", "Email")

All but the last part of the key must not contain periods. In this example, only Email may contain periods.

The key "671314941988.test@ausocean.org" would match 67131494198 or "test@ausocean.org" or both.

To match both Skey and Email:

q.Filter("Skey =", 671314941988)
q.Filter("Email =", "test@ausocean.org)

To match just Skey, i.e., return all entities for a given site, simply omit the Email part as follows.

q.Filter("Skey =", 671314941988)

Alternatively, specify nil for the Email which is a wild card.

q.Filter("Skey =", 671314941988)
q.Filter("Email =", nil)

Using nil is most useful however when ignoring a key part which has key parts before and after that much be matched.

To match just Email, i.e., return all entities for a given user.

q.Filter("Email =", "test@ausocean.org)

The following query would however fail due to the wrong order:

q.Filter("Email =", "test@ausocean.org)
q.Filter("Skey =", 671314941988)

FileStore represents all keys as strings. To faciliate substring comparisons, 64-bit ID keys are represented as two 32-bit integers separated by a period and the keyParts in NewQuery must reflect this.

func (*FileStore) Create

func (s *FileStore) Create(ctx context.Context, key *Key, src Entity) error

func (*FileStore) DeleteMulti

func (s *FileStore) DeleteMulti(ctx context.Context, keys []*Key) error

func (*FileStore) Get

func (s *FileStore) Get(ctx context.Context, key *Key, dst Entity) error

Get returns a single entity by its key.

func (*FileStore) GetAll

func (s *FileStore) GetAll(ctx context.Context, query Query, dst interface{}) ([]*Key, error)

GetAll implements Store.GetAll for FileStore. Unlike CloudStore queries are only valid against key parts. If the entity has a property named Key, it is automatically populated with its Key value.

func (*FileStore) IDKey

func (s *FileStore) IDKey(kind string, id int64) *Key

IDKey returns a FileStore ID key for the given kind, setting Name to the file name. A 64-bit ID is represented as two 32-bit unsigned integers separated by a period.

func (*FileStore) NameKey

func (s *FileStore) NameKey(kind, name string) *Key

NameKey returns a FileStore name key for the given kind, setting Name to the file name.

func (*FileStore) NewQuery

func (s *FileStore) NewQuery(kind string, keysOnly bool, keyParts ...string) Query

NewQuery creates and returns a new FileQuery.

func (*FileStore) Put

func (s *FileStore) Put(ctx context.Context, key *Key, src Entity) (*Key, error)

func (*FileStore) Update

func (s *FileStore) Update(ctx context.Context, key *Key, fn func(Entity), dst Entity) error

type Key

type Key = datastore.Key // Datastore key.

We reuse some Google Datastore types.

type Property

type Property = datastore.Property // Datastore property.

We reuse some Google Datastore types.

type Query

type Query interface {
	Filter(filterStr string, value interface{}) error // Filters a query.
	Order(fieldName string)                           // Orders a query.
	Limit(limit int)                                  // Limits the number of results returned.
}

Query defines the query interface, which is a subset of Google Cloud's datastore.Query. Note that unlike datastore.Query methods, these methods modify the Query value without returning it. A nil value denotes a wild card (match anything).

See also Google Cloud datastore.Query.Filter and datastore.Query.Order.

type Store

type Store interface {
	IDKey(kind string, id int64) *Key                                        // Returns an ID key.
	NameKey(kind, name string) *Key                                          // Returns a name key.
	NewQuery(kind string, keysOnly bool, keyParts ...string) Query           // Returns a new query.
	Get(ctx context.Context, key *Key, dst Entity) error                     // Gets a single entity by its key.
	GetAll(ctx context.Context, q Query, dst interface{}) ([]*Key, error)    // Runs a query and returns all matching entities.
	Create(ctx context.Context, key *Key, src Entity) error                  // Creates a single entity by its key.
	Put(ctx context.Context, key *Key, src Entity) (*Key, error)             // Put or creates a single entity by its key.
	Update(ctx context.Context, key *Key, fn func(Entity), dst Entity) error // Atomically updates a single entity by its key.
	DeleteMulti(ctx context.Context, keys []*Key) error                      // Deletes multiple entities by their keys.
}

Store defines the datastore interface. It is a blend and subset of Google Cloud datastore functions and datastore.Client methods.

See also https://godoc.org/cloud.google.com/go.

func NewStore

func NewStore(ctx context.Context, kind, id, url string) (s Store, err error)

NewStore returns a new Store. If kind is "cloud" a CloudStore is returned. If kind is "file" a FileStore is returned. The id is the Project ID of the requested datastore service. The url parameter is a URL to credentials for CloudStore, or a directory path for FileStore.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL