ddb

package module
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 2, 2022 License: MIT Imports: 16 Imported by: 44

README

ddb

Go Reference

Common Fate helpers for working with DynamoDB.

Integration testing

You can provision an example table for testing as follows.

go run cmd/create/main.go

To run the integration tests, you need to set the TESTING_DYNAMODB_TABLE to be the name of the test table you created.

export TESTING_DYNAMODB_TABLE=ddb-testing

To cleanup the table:

go run cmd/destroy/main.go

Documentation

Overview

Example (CustomUnmarshalling)

For complex queries you can implement UnmarshalQueryOutput to control how the DynamoDB query results are unmarshaled. 

package main

import (
	"context"
	"fmt"

	"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb"
	"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
	"github.com/common-fate/ddb"
)

type Car struct{}

type Wheel struct{}

// Result is an example complex query result for the example access pattern.
// In this example, we fetch both a Car object as well as it's associated Wheels
// in the same query
type Result struct {
	Car    Car
	Wheels []Wheel
}

type ListCarAndWheelsByColor struct {
	Color  string
	Result Result
}

func (l *ListCarAndWheelsByColor) BuildQuery() (*dynamodb.QueryInput, error) {
	// the empty QueryInput is just for the example.
	// in a real query this wouldn't be empty.
	return &dynamodb.QueryInput{}, nil
}

func (l *ListCarAndWheelsByColor) UnmarshalQueryOutput(out *dynamodb.QueryOutput) error {
	// an example of custom unmarshalling logic for complex queries which return multiple item types
	for _, item := range out.Items {
		typeField, ok := item["type"].(*types.AttributeValueMemberS)
		if !ok {
			return fmt.Errorf("couldn't unmarshal: %+v", item)
		}

		if typeField.Value == "car" {
			err := attributevalue.UnmarshalMap(item, &l.Result.Car)
			if err != nil {
				return err
			}
		} else {
			var wheel Wheel
			err := attributevalue.UnmarshalMap(item, &wheel)
			if err != nil {
				return err
			}
			l.Result.Wheels = append(l.Result.Wheels, wheel)
		}
	}
	return nil
}

// For complex queries you can implement UnmarshalQueryOutput to control how
// the DynamoDB query results are unmarshaled.
func main() {
	ctx := context.TODO()

	q := ListCarAndWheelsByColor{Color: "light-orange"}
	c, _ := ddb.New(ctx, "example-table")
	_, _ = c.Query(ctx, &q, nil)

	// q.Result.Car and q.Result.Wheels are now populated with data as fetched from DynamoDB
}

Output:
Example (Simple)

For simple queries you can declare the query as a type alias. ddb will unmarshal the results directly into the query struct, as shown below. 

package main

import (
	"context"

	"github.com/aws/aws-sdk-go-v2/service/dynamodb"
	"github.com/common-fate/ddb"
)

// Apple is an example object which we will
// show how to read from the database with some access patterns.
type Apple struct{}

type ListApples []Apple

func (l *ListApples) BuildQuery() (*dynamodb.QueryInput, error) {
	// the empty QueryInput is just for the example.
	// in a real query this wouldn't be empty.
	return &dynamodb.QueryInput{}, nil
}

// For simple queries you can declare the query as a type alias.
// ddb will unmarshal the results directly into the query struct, as shown below.
func main() {
	ctx := context.TODO()

	var la ListApples
	c, _ := ddb.New(ctx, "example-table")
	_, _ = c.Query(ctx, &la, nil)

	// la is now populated with []Apple as fetched from DynamoDB
}

Output:
Example (StructTag) 
package main

import (
	"context"

	"github.com/aws/aws-sdk-go-v2/service/dynamodb"
	"github.com/common-fate/ddb"
)

// Orange is an example object which we will
// show how to read from the database with some access patterns.
type Orange struct{}

type ListOrangesByColor struct {
	Color  string
	Result []Orange `ddb:"result"` // results will be unmarshaled into this field.
}

func (l *ListOrangesByColor) BuildQuery() (*dynamodb.QueryInput, error) {
	// the empty QueryInput is just for the example.
	// in a real query this wouldn't be empty.
	return &dynamodb.QueryInput{}, nil
}

// For queries that take parameters (like an object ID or a status), using the `ddb:"result"`
// struct tag is the simplest way to denote the results field of the query.
// ddb will unmarshal the results into the tagged field.

func main() {
	ctx := context.TODO()

	lobc := ListOrangesByColor{Color: "light-orange"}
	c, _ := ddb.New(ctx, "example-table")
	_, _ = c.Query(ctx, &lobc, nil)

	// labc.Result is now populated with []Orange as fetched from DynamoDB
}

Output:
Example (Transaction)

ddb exposes a transactions API. 

package main

import (
	"context"

	"github.com/common-fate/ddb"
)

type Item struct {
	ID string
}

func (i Item) DDBKeys() (ddb.Keys, error) {
	k := ddb.Keys{
		PK: i.ID,
		SK: "SK",
	}
	return k, nil
}

// ddb exposes a transactions API.
func main() {
	ctx := context.TODO()

	c, _ := ddb.New(ctx, "example-table")
	tx := c.NewTransaction()

	tx.Put(Item{ID: "1"})
	tx.Put(Item{ID: "2"})
	tx.Delete(Item{ID: "3"})
	_ = tx.Execute(ctx)
}

Output:

Index

Examples

Constants

This section is empty.

Variables

View Source 
var ErrInvalidBatchSize error = errors.New("batch size must be greater than 0 and must not be greater than 25")

ErrInvalidBatchSize is returned if an invalid batch size is specified when creating a ddb instance.

View Source 
var ErrNoEntityType = errors.New("item does not have a 'ddb:type' field")
View Source 
var ErrNoItems error = errors.New("item query returned no items")

ErrNoItems is returned when we expect a query result to contain items, but it doesn't contain any.

Functions

func ConsistentRead added in v0.15.0 

func ConsistentRead() func(*QueryOpts)

ConsistentRead enables strong read consistency. Strongly consistent reads are not supported on global secondary indexes. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html

func GetConsistentRead added in v0.15.0 

func GetConsistentRead(enabled bool) func(*GetOpts)

GetConsistentRead customises strong read consistency. By default, Get() uses consistent reads. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html

func GetItemEntityType added in v0.15.0 

func GetItemEntityType(item map[string]types.AttributeValue) (string, error)

GetItemEntityType gets the entity type of a raw DynamoDB item. The item must have a 'ddb:type' field on it. If it doesn't, an error is returned.

To use this, implement ddb.EntityTyper on your objects.

func Limit added in v0.8.0 

func Limit(limit int32) func(*QueryOpts)

Limit overrides the amount of items returned from the query. It is mapped to the 'Limit' argument in the dynamodb.Query method.

func Page added in v0.8.0 

func Page(pageToken string) func(*QueryOpts)

Page sets the pagination token to provide an offset for the query. It is mapped to the 'ExclusiveStartKey' argument in the dynamodb.Query method.

func WithBatchSize added in v0.7.0 

func WithBatchSize(batchSize int) func(*Client)

WithBatchSize allows a custom batchSize to be provided for putBatch operations.

func WithDynamoDBClient added in v0.4.0 

func WithDynamoDBClient(d *dynamodb.Client) func(*Client)

WithDynamoDBClient allows a custom dynamodb.Client to be provided.

func WithPageTokenizer added in v0.8.0 

func WithPageTokenizer(e Tokenizer) func(*Client)

WithPageTokenizer allows a tokenizer to be provided for turning LastEvaluatedKey items into strings.

Types

type Client added in v0.4.0 

type Client struct {
	// contains filtered or unexported fields
}

Client is a thin wrapper over the native DynamoDB client. It has methods which allow access patterns to be written in a more ergonomic fashion than the native client.

func New added in v0.4.0 

func New(ctx context.Context, table string, opts ...func(*Client)) (*Client, error)

New creates a new DynamoDB Client.

func (*Client) Client added in v0.15.0 

func (c *Client) Client() *dynamodb.Client

func (*Client) Delete added in v0.7.0 

func (c *Client) Delete(ctx context.Context, item Keyer) error

Delete calls DeleteItem to delete an item in DynamoDB.

func (*Client) DeleteBatch added in v0.13.0 

func (c *Client) DeleteBatch(ctx context.Context, items ...Keyer) error

DeleteBatch calls BatchWriteItem to create or update items in DynamoDB.

DynamoDB BatchWriteItem api has a limit of 25 items per batch. DeleteBatch will automatically split the items into batches of 25 by default.

You can override this default batch size using WithBatchSize(n) when you initialize the client.

func (*Client) Get added in v0.15.0 

func (c *Client) Get(ctx context.Context, key GetKey, item Keyer, opts ...func(*GetOpts)) (*GetItemResult, error)

Get calls GetItem to get an item in DynamoDB. Get defaults to using consistent reads.

func (*Client) NewTransaction added in v0.15.0 

func (c *Client) NewTransaction() Transaction

func (*Client) Put added in v0.4.0 

func (c *Client) Put(ctx context.Context, item Keyer) error

Put calls PutItem to create or update an item in DynamoDB.

func (*Client) PutBatch added in v0.4.0 

func (c *Client) PutBatch(ctx context.Context, items ...Keyer) error

PutBatch calls BatchWriteItem to create or update items in DynamoDB.

DynamoDB BatchWriteItem api has a limit of 25 items per batch. PutBatch will automatically split the items into batches of 25 by default.

You can override this default batch size using WithBatchSize(n) when you initialize the client.

func (*Client) Query added in v0.4.0 

func (c *Client) Query(ctx context.Context, qb QueryBuilder, opts ...func(*QueryOpts)) (*QueryResult, error)

Query DynamoDB using a given QueryBuilder. Under the hood, this uses the QueryItems API.

The QueryBuilder 'qb' defines the query, as well as how to unmarshal the result back into Go objects. The unmarshaling logic works as follows:

1. If qb implements UnmarshalQueryOutput, call it and return.

2. If qb contains a field with a `ddb:"result"` struct tag, unmarshal results to that field.

3. Unmarshal the results directly to qb.

The examples in this package show how to write simple and complex access patterns which use each of the three methods above.

func (*Client) Table added in v0.15.0 

func (c *Client) Table() string

func (*Client) TransactWriteItems added in v0.6.0 

func (c *Client) TransactWriteItems(ctx context.Context, tx []TransactWriteItem) error

type DBTransaction added in v0.15.0 

type DBTransaction struct {
	// contains filtered or unexported fields
}

DBTransaction writes pending items to slices in memory. It is goroutine-safe.

func (*DBTransaction) Delete added in v0.15.0 

func (t *DBTransaction) Delete(item Keyer)

func (*DBTransaction) Execute added in v0.15.0 

func (t *DBTransaction) Execute(ctx context.Context) error

func (*DBTransaction) Put added in v0.15.0 

func (t *DBTransaction) Put(item Keyer)

type EntityTyper added in v0.15.0 

type EntityTyper interface {
	EntityType() string
}

If EntityType is implemented, ddb will write a special 'ddb:type' when marshalling the object. This can be used to implement custom unmarshalling for queries which return multiple object types.

For example, you may wish to save an invoice along with its line items as separate rows in DynamoDB. The `EntityType` of the invoice can be "invoice", and then `EntityType` of the line item can be "lineItem". When querying the database and unmarshalling these objects back into Go structs, you can check the type of them by looking at the value of 'ddb:type'.

type GetItemResult added in v0.15.0 

type GetItemResult struct {
	// RawOutput is the DynamoDB API response. Usually you won't need this,
	// as results are parsed onto the item argument.
	RawOutput *dynamodb.GetItemOutput
}

type GetKey added in v0.15.0 

type GetKey struct {
	PK string
	SK string
}

GetKey is the key of the item to get. The GetItem API always uses the primary key.

type GetOpts added in v0.15.0 

type GetOpts struct {
	ConsistentRead bool
}

type JSONTokenizer added in v0.8.0 

type JSONTokenizer struct{}

func (*JSONTokenizer) MarshalToken added in v0.8.0 

func (e *JSONTokenizer) MarshalToken(ctx context.Context, item map[string]types.AttributeValue) (string, error)

func (*JSONTokenizer) UnmarshalToken added in v0.8.0 

func (e *JSONTokenizer) UnmarshalToken(ctx context.Context, s string) (map[string]types.AttributeValue, error)

type KMSTokenizer added in v0.10.0 

type KMSTokenizer struct {
	// contains filtered or unexported fields
}

func NewKMSTokenizer added in v0.10.0 

func NewKMSTokenizer(ctx context.Context, key string) (*KMSTokenizer, error)

func (*KMSTokenizer) MarshalToken added in v0.10.0 

func (e *KMSTokenizer) MarshalToken(ctx context.Context, item map[string]types.AttributeValue) (string, error)

func (*KMSTokenizer) UnmarshalToken added in v0.10.0 

func (e *KMSTokenizer) UnmarshalToken(ctx context.Context, s string) (map[string]types.AttributeValue, error)

type Keyer added in v0.4.0 

type Keyer interface {
	DDBKeys() (Keys, error)
}

Keyers give DynamoDB keys to be used when inserting an item.

type Keys added in v0.4.0 

type Keys struct {
	PK     string
	SK     string
	GSI1PK string `json:",omitempty"`
	GSI1SK string `json:",omitempty"`
	GSI2PK string `json:",omitempty"`
	GSI2SK string `json:",omitempty"`
	GSI3PK string `json:",omitempty"`
	GSI3SK string `json:",omitempty"`
	GSI4PK string `json:",omitempty"`
	GSI4SK string `json:",omitempty"`
}

Keys are primary and Global Secondary Index (GSI) keys to be used when storing an item in DynamoDB. The ddb package is opinionated on the naming of these keys.

type QueryBuilder added in v0.4.0 

type QueryBuilder interface {
	BuildQuery() (*dynamodb.QueryInput, error)
}

QueryBuilders build query inputs for DynamoDB access patterns. The inputs are passed to the QueryItems DynamoDB API.

When writing a new QueryBuilder access pattern you should always implement integration tests for it against a live DynamoDB database.

type QueryOpts added in v0.8.0 

type QueryOpts struct {
	PageToken      string
	Limit          int32
	ConsistentRead bool
}

type QueryOutputUnmarshaler added in v0.4.0 

type QueryOutputUnmarshaler interface {
	UnmarshalQueryOutput(out *dynamodb.QueryOutput) error
}

QueryOutputUnmarshalers implement custom logic to unmarshal the results of a DynamoDB QueryItems call.

type QueryResult added in v0.8.0 

type QueryResult struct {
	// RawOutput is the DynamoDB API response. Usually you won't need this,
	// as results are parsed onto the QueryBuilder argument.
	RawOutput *dynamodb.QueryOutput

	// NextPage is the next page token. If empty, there is no next page.
	NextPage string
}

type Storage added in v0.4.0 

type Storage interface {
	Query(ctx context.Context, qb QueryBuilder, opts ...func(*QueryOpts)) (*QueryResult, error)
	Put(ctx context.Context, item Keyer) error
	PutBatch(ctx context.Context, items ...Keyer) error
	TransactWriteItems(ctx context.Context, tx []TransactWriteItem) error
	NewTransaction() Transaction
	Delete(ctx context.Context, item Keyer) error
	DeleteBatch(ctx context.Context, items ...Keyer) error
	// Get performs a GetItem call to fetch a single item from DynamoDB.
	// The results are written to the 'item' argument. This argument
	// must be passed by reference to the method.
	//
	// 	var item MyItem
	//	db.Get(ctx, ddb.GetKey{PK: ..., SK: ...}, &item)
	Get(ctx context.Context, key GetKey, item Keyer, opts ...func(*GetOpts)) (*GetItemResult, error)
	// Client returns the underlying DynamoDB client. It's useful for cases
	// where you need more control over queries or writes than the ddb library provides.
	Client() *dynamodb.Client
	// Table returns the name of the DynamoDB table that the client is configured to use.
	Table() string
}

Storage defines a common interface to make testing ddb easier. Both the real and mock clients meet this interface.

type Tokenizer added in v0.8.0 

type Tokenizer interface {
	MarshalToken(ctx context.Context, item map[string]types.AttributeValue) (string, error)
	UnmarshalToken(ctx context.Context, s string) (map[string]types.AttributeValue, error)
}

Tokenizer converts DynamoDB page cursor items to and from opaque strings.

type TransactWriteItem added in v0.6.0 

type TransactWriteItem struct {
	Put    Keyer
	Delete Keyer
}

TransactWriteItem is a wrapper over the DynamoDB TransactWriteItem type. Currently, only the Put option is exposed. The API supports other operations which can be added to this struct.

see: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html

type Transaction added in v0.15.0 

type Transaction interface {
	// Put adds an item to be written in the transaction.
	Put(item Keyer)
	// Delete adds a item to be delete in the transaction.
	Delete(item Keyer)
	// Execute the transaction.
	// This calls the TransactWriteItems API.
	// See: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactWriteItems.html
	Execute(ctx context.Context) error
}

Transactions allow atomic write operations to be made to a DynamoDB table. DynamoDB transactions support up to 100 operations.

Calling Put() and Delete() on a transaction register items in memory to be written to the table. No API calls are performed until Execute() is called.

Source Files

View all Source files

Directories

Path Synopsis
cmd
create
destroy

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.