txnBuf

Documentation

Overview

Package txnBuf contains a transaction buffer filter for the datastore service.

By default, datastore transactions take a snapshot of the entity group as soon as you Get or Put into it. All subsequent Get (and query) operations reflect the state of the ORIGINAL transaction snapshot, regardless of any Put/Delete operations you've done since the beginning of the transaction.

If you've installed this transaction buffer, then:

• All mutations will be reflected in all read operations (see LIMITATIONS). Without the buffer, read operations always observe the state of the entity group(s) at the time that the transaction started.

• All mutation operations will be buffered until the close of the transaction. This can help reduce the transaction size, and thus avoid the transaction size limit (currently 10MB). Multiple puts to the same entity will not increase the transaction size multiple times.

• Transactions inside of an existing transaction will add to their outer transaction if they don't cause the outer transaction to exceed its size budget.

• If an inner transaction would cause the OUTERMOST transaction to exceed the appengine-imposed 10MB transaction size limit, an error will be returned from the inner transaction, instead of adding it into the outermost transaction. This only applies to the first level of inner transactions, and does not apply to recursive transactions. The reason for this is that it's entirely feasible for inner transactions to temporarially exceed the limit, but still only commit an outer transaction which is under the limit. An example of this would be having one inner-inner transaction add a lot of large entities and then having a subsequent inner-inner transaction delete some of those entities.

LIMITATIONS (only inside of a transaction)

• KeysOnly/Projection/Count queries are supported, but may incur additional costs.

  These query types are implemented via projection queries, but will project all order-by fields in addition to any specified in the original query.

• Distinct Projection queries do all 'distinct' deduplication in-memory. This could make them substantially more expensive than their native equivalent.

• Metadata entities (e.g. `__entity_group__`) will reflect their values as they were at the beginning of the transaction, and will not increment as you write inside of the transaction.

• Query cursors are not supported. Since the cursor format for the in-memory datastore implementation isn't compatible with the production cursors, it would be pretty tricky to make it so that cursors were viable outside the transaction as well as inside of it while also having it accurately reflect the 'merged' query results.

• No parallel access* to datastore while in a transaction; all nested operations are serialized. This is done for simplicity and correctness.

• The exception is that callbacks inside of a Run/GetMulti/DeleteMulti/PutMulti query MAY read/write the current transaction. Modifications to the datastore during query executions will not affect the query results (e.g. the query has snapshot consistency from the moment that it begins iteration). Note, however, that datastore operations within the callback are still synchronized. This behavior is so that the user is not forced to buffer all of the query results before doing work with them, but can treat the query like a stream of events, if they so choose.

• The changing of namespace inside of a transaction is undefined... This is just generally a terrible idea anyway, but I thought it was worth mentioning.

• Currently, the soft transactions are not directly accessible using the CurrentTransaction interface; it returns the wrapped datastore's transaction. While this is still correct, it could definitely be made more useful by adding transaction buffer metadata to the returned object.

Index

• Constants
• Variables
• func FilterRDS(c context.Context) context.Context

Constants

const DefaultSizeBudget = int64((10 * 1000 * 1000) * 0.95)

DefaultSizeBudget is the size budget for the root transaction.

Because our estimation algorithm isn't entirely correct, we take 5% off the limit for encoding and estimate inaccuracies.

10MB taken on 2015/09/24: https://cloud.google.com/appengine/docs/go/datastore/#Go_Quotas_and_limits

const DefaultWriteCountBudget = 500

DefaultWriteCountBudget is the maximum number of entities that can be written in a single call.

This is not known to be documented, and has instead been extracted from a datastore error message.

const XGTransactionGroupLimit = 25

XGTransactionGroupLimit is the number of transaction groups to allow in an XG transaction.

25 taken on 2015/09/24: https://cloud.google.com/appengine/docs/go/datastore/transactions#Go_What_can_be_done_in_a_transaction

Variables

var ErrTooManyRoots = errors.New(
	"operating on too many entity groups in nested transaction")

ErrTooManyRoots is returned when executing an operation which would cause the transaction to exceed it's allotted number of entity groups.

var ErrTransactionTooLarge = errors.New(
	"applying the transaction would make the parent transaction too large")

ErrTransactionTooLarge is returned when applying an inner transaction would cause an outer transaction to become too large.

Functions

func FilterRDS

func FilterRDS(c context.Context) context.Context

FilterRDS installs a transaction buffer datastore filter in the context.