dynastorev2

README

dynastorev2

This package provides a CRUD (create, read, update and delete) store for Amazon DynamoDB using the AWS Go SDK v2.

Overview

This is a rewrite of the original dynastore with the main differences being:

1. It uses the Generics feature added in Go 1.18
2. It is built on AWS Go SDK v2.
3. The API has been simplified.
4. I am still learning how to use Generics in Go...

Example

	ctx := context.Background()

	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		// handle error
	}

	client = dynamodb.NewFromConfig(cfg)
	customerStore := dynastorev2.New[string, string, []byte](client, "tickets-table")

	fields := map[string]any{
		"created": time.Now().UTC().Round(time.Millisecond),
	}

	res, err := customerStore.Create(ctx,
		"customer",                                 // partition key
		"01FCFSDXQ8EYFCNMEA7C2WJG74",               // sort key
		[]byte(`{"name": "Stax"}`),                 // value, in this case JSON encoded value
		customerStore.WriteWithExtraFields(fields), // extra fields which could be indexed in the future
	)
	if err != nil {
		// handle error
	}

	// print out the version from the mutation result, this is used for optimistic locking
	fmt.Println("version", res.Version)

Creates a record which looks like this in Amazon DynamoDB.

id (Partition Key)	name (Sort Key)	version	payload	expires	created
customer	01FCFSDXQ8EYFCNMEA7C2WJG74	1	{"name": "Stax"}	null	2022-04-10T06:27:16.994Z

Note: This library doesn't aim to provide a high level abstraction for Amazon DynamoDB, you will need to learn how it works to understand some of the limitations to use it successfully.

Implementation Tips

Before you get started i recommend you review some of the code examples provided in the Amazon DynamoDB documentation.

1. Don't use Amazon DynamoDB if you have anything beyond a simple K/V compatible model until you understand your access patterns.
2. Use a single-table design to model your data.
3. Use Universally Unique Lexicographically Sortable Identifier (ULID) for sort keys, this will help ensure a rational order of data in the table. Sort key by default is sorted in descending order, oldest first, newest last, exploiting this behaviour may mitigate some of the limitations with Amazon DynamoDB.
4. If your using WriteWithTTL you need to deal with the fact that Amazon DynamoDB doesn't delete expired data straight away, records can hang around for up to 48 hours according to the documentation.

Status

• Added CRUD with conditional checks and tests
• List with pagination
• Optimistic Locking for Updates
• Locking
• Leasing

References

Prior work in this space:

• https://github.com/wolfeidau/dynastore
• https://github.com/wolfeidau/dynalock
• https://github.com/awslabs/dynamodb-lock-client
• https://github.com/intercom/lease

Updates to the original API are based on a great blog post by @davecheney https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis

License

This code was authored by Mark Wolfe and licensed under the Apache 2.0 license.

Documentation

Index

• Constants
• Variables
• type DeleteOption
• type Key
• type OperationDetails
  • func OperationDetailsFromContext(ctx context.Context) *OperationDetails
• type OperationResult
• type ReadOption
• type Store
  • func New[P Key, S Key, V any](client *dynamodb.Client, tableName string, options ...StoreOption[P, S, V]) *Store[P, S, V]
  • func (t *Store[P, S, V]) Create(ctx context.Context, partitionKey P, sortKey S, value V, ...) (*OperationResult, error)
  • func (t *Store[P, S, V]) Delete(ctx context.Context, partitionKey P, sortKey S, options ...DeleteOption[P, S]) error
  • func (t *Store[P, S, V]) DeleteWithCheck(enabled bool) DeleteOption[P, S]
  • func (t *Store[P, S, V]) Get(ctx context.Context, partitionKey P, sortKey S, options ...ReadOption[P, S]) (*OperationResult, V, error)
  • func (t *Store[P, S, V]) ListBySortKeyPrefix(ctx context.Context, partitionKey P, prefix string, ...) (*OperationResult, []V, error)
  • func (t *Store[P, S, V]) ReadWithConsistentRead(consistentRead bool) ReadOption[P, S]
  • func (t *Store[P, S, V]) ReadWithLastEvaluatedKey(lastEvaluatedKey string) ReadOption[P, S]
  • func (t *Store[P, S, V]) ReadWithLimit(limit int32) ReadOption[P, S]
  • func (t *Store[P, S, V]) Update(ctx context.Context, partitionKey P, sortKey S, value V, ...) (*OperationResult, error)
  • func (t *Store[P, S, V]) WriteWithCreateConstraintDisabled(createConstraintDisabled bool) WriteOption[P, S, V]
  • func (t *Store[P, S, V]) WriteWithExtraFields(extraFields map[string]any) WriteOption[P, S, V]
  • func (t *Store[P, S, V]) WriteWithTTL(ttl time.Duration) WriteOption[P, S, V]
  • func (t *Store[P, S, V]) WriteWithVersion(version int64) WriteOption[P, S, V]
• type StoreHooks
• type StoreOption
  • func WithStoreHooks[P Key, S Key, V any](storeHooks *StoreHooks[P, S, V]) StoreOption[P, S, V]
• type WriteOption

Constants

const (
	// DefaultPartitionKeyAttribute this is the default partition key attribute name
	DefaultPartitionKeyAttribute = "id"

	// DefaultSortKeyAttribute this is the default sort key attribute name
	DefaultSortKeyAttribute = "name"

	// DefaultExpiresAttribute this is the default name for the dynamodb expiration attribute
	DefaultExpiresAttribute = "expires"

	// DefaultVersionAttribute this is the default name for the dynamodb version attribute used for optimistic locking during creates and atomic updates
	DefaultVersionAttribute = "version"

	// DefaultPayloadAttribute this is the default attribute name containing the encoded payload of the record
	DefaultPayloadAttribute = "payload"
)

Variables

var (
	// ErrReservedField extra fields provided have an entry which conflicts with the keys in the table
	ErrReservedField = errors.New("dynastorev2: extra fields contained name which conflicts with table keys attributes")

	// ErrDeleteFailedKeyNotExists delete failed due to constraint added which checks the record exists when deleting
	ErrDeleteFailedKeyNotExists = errors.New("dynastorev2: delete failed as the partition and sort keys didn't exist in the table")

	// ErrKeyNotExists get failed due to partition and sort keys didn't exist in the table
	ErrKeyNotExists = errors.New("dynastorev2: get failed as the partition and sort keys didn't exist in the table")
)

Types

type DeleteOption

type DeleteOption[P Key, S Key] interface {
	// contains filtered or unexported methods
}

DeleteOption sets a specific delete option

type Key

type Key interface {
	string | constraints.Integer | []byte
}

Key ensures the partition or sort key used is a valid type for DynamoDB, note this is also referred to as the Primary Key in the AWS documentation.

Each key attribute must be a scalar (meaning that it can hold only a single value).

type OperationDetails

type OperationDetails struct {
	Name         string
	PartitionKey string
	SortKey      string
}

func OperationDetailsFromContext

func OperationDetailsFromContext(ctx context.Context) *OperationDetails

OperationName extracts the name of the operation being handled in the given context. If it is not known, it returns nil.

type OperationResult

type OperationResult struct {
	Version          int64                   `json:"version,omitempty"`
	ConsumedCapacity *types.ConsumedCapacity `json:"consumed_capacity,omitempty"`
	LastEvaluatedKey string                  `json:"last_evaluated_key,omitempty"`
}

OperationResult returned with operations to provide some information about the update

type ReadOption

type ReadOption[P Key, S Key] interface {
	// contains filtered or unexported methods
}

readOptions sets a specific read option

type Store

type Store[P Key, S Key, V any] struct {
	// contains filtered or unexported fields
}

Store store using aws sdk v2

func New

func New[P Key, S Key, V any](client *dynamodb.Client, tableName string, options ...StoreOption[P, S, V]) *Store[P, S, V]

New creates and configures a new store using aws sdk v2

func (*Store[P, S, V]) Create

func (t *Store[P, S, V]) Create(ctx context.Context, partitionKey P, sortKey S, value V, options ...WriteOption[P, S, V]) (*OperationResult, error)

Create a record in DynamoDB using the provided partition and sort keys, a payload containing the value

Note this will use a condition to ensure the specified partition and sort keys don't exist in DynamoDB.

func (*Store[P, S, V]) Delete

func (t *Store[P, S, V]) Delete(ctx context.Context, partitionKey P, sortKey S, options ...DeleteOption[P, S]) error

Delete a record in DynamoDB using the provided partition and sort keys

func (*Store[P, S, V]) DeleteWithCheck

func (t *Store[P, S, V]) DeleteWithCheck(enabled bool) DeleteOption[P, S]

DeleteWithCheck delete with a check condition to ensure the record exists

func (*Store[P, S, V]) Get

func (t *Store[P, S, V]) Get(ctx context.Context, partitionKey P, sortKey S, options ...ReadOption[P, S]) (*OperationResult, V, error)

Get a record in DynamoDB using the provided partition and sort keys

func (*Store[P, S, V]) ListBySortKeyPrefix

func (t *Store[P, S, V]) ListBySortKeyPrefix(ctx context.Context, partitionKey P, prefix string, options ...ReadOption[P, S]) (*OperationResult, []V, error)

ListBySortKeyPrefix perform a query of the DynamoDB using hte partition key and a string prefix for the sort key. This is typically used when hierarchies are stored in this partition. For example if we have a customer addresses with an sort key with a format of (customer id)/(address id), to list the addresses for a customer you list using the customer id as the prefix.

Notes: 1. You the sort key must be a string to support this operation, this is a limitation of the AWs SDK. 2. ListBySortKeyPrefix will also return expired records as these may hang around for up to 48 hours according to the documentation, see: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html

func (*Store[P, S, V]) ReadWithConsistentRead

func (t *Store[P, S, V]) ReadWithConsistentRead(consistentRead bool) ReadOption[P, S]

ReadWithConsistentRead enable the consistent read flag when performing get operations

func (*Store[P, S, V]) ReadWithLastEvaluatedKey

func (t *Store[P, S, V]) ReadWithLastEvaluatedKey(lastEvaluatedKey string) ReadOption[P, S]

ReadWithLastEvaluatedKey provide a last evaluated key when performing list operations

func (*Store[P, S, V]) ReadWithLimit

func (t *Store[P, S, V]) ReadWithLimit(limit int32) ReadOption[P, S]

ReadWithLimit provide a record count limit when performing list operations

func (*Store[P, S, V]) Update

func (t *Store[P, S, V]) Update(ctx context.Context, partitionKey P, sortKey S, value V, options ...WriteOption[P, S, V]) (*OperationResult, error)

Update a record in DynamoDB using the provided partition and sort keys, a payload containing the value

Note this will use a condition to ensure the specified partition and sort keys exist in DynamoDB.

func (*Store[P, S, V]) WriteWithCreateConstraintDisabled

func (t *Store[P, S, V]) WriteWithCreateConstraintDisabled(createConstraintDisabled bool) WriteOption[P, S, V]

WriteWithCreateConstraintDisabled disable the check on create for existence of the rows

func (*Store[P, S, V]) WriteWithExtraFields

func (t *Store[P, S, V]) WriteWithExtraFields(extraFields map[string]any) WriteOption[P, S, V]

WriteWithExtraFields assign extra fields provided to the record when written or updated

func (*Store[P, S, V]) WriteWithTTL

func (t *Store[P, S, V]) WriteWithTTL(ttl time.Duration) WriteOption[P, S, V]

WriteWithTTL assigns a time to live (TTL) to the record when it is created or updated

func (*Store[P, S, V]) WriteWithVersion

func (t *Store[P, S, V]) WriteWithVersion(version int64) WriteOption[P, S, V]

WriteWithVersion adds a condition check the provided version to enable optimistic locking

type StoreHooks

type StoreHooks[P Key, S Key, V any] struct {
	// RequestBuilt will be invoked prior to dispatching the request to the AWS SDK
	RequestBuilt     func(ctx context.Context, pk P, sk S, params any) context.Context
	ResponseReceived func(ctx context.Context, pk P, sk S, params any) context.Context
}

StoreHooks is a container for callbacks that can instrument the datastore

type StoreOption

type StoreOption[P Key, S Key, V any] interface {
	// contains filtered or unexported methods
}

StoreOption sets a specific store option

func WithStoreHooks

func WithStoreHooks[P Key, S Key, V any](storeHooks *StoreHooks[P, S, V]) StoreOption[P, S, V]

type WriteOption

type WriteOption[P Key, S Key, V any] interface {
	// contains filtered or unexported methods
}

Option sets a specific write option