autoquery

package module
v0.0.0-...-257e3d0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 4, 2022 License: MIT Imports: 11 Imported by: 0

README

dynamodb-autoquery

DynamoDB querying with automatic index selection and a friendly interface.

Installation

go get -v github.com/dgravesa/dynamodb-autoquery

About

autoquery leverages table index metadata provided through a DescribeTable call (or optionally, another metadata provider) to automatically select a query index based on viability and scoring against a particular query expression.

Executing a query

  1. Initialize an autoquery client instance using a DynamoDB service instance.
svc := dynamodb.New(session.New())
client := autoquery.NewClient(svc)
  1. Build an expression with any desired filter conditions.
expr := autoquery.Key("director").Equal("Clint Eastwood").
    And("title").BeginsWith("The ").
    OrderBy("rating", false). // high to low
    Select("title", "year", "rating")
  1. Apply the expression on a table to initialize the query result parser.
tableName := "Movies"
parser := client.Query(tableName, expr)
  1. Parse result items.
var movie struct {
    Title  string  `dynamodbav:"title"`
    Year   int     `dynamodbav:"year"`
    Rating float64 `dynamodbav:"rating"`
}

var err error
maxPrintCount := 10
for count := 0; count < maxPrintCount; count++ {
    // parse movie items until max count is reached or all have been parsed
    err = parser.Next(context.Background(), &movie)
    if err != nil {
        break
    }

    fmt.Printf("Year: %d, Title: %s, Rating: %.1f\n", movie.Year, movie.Title, movie.Rating)
}

// check error type
switch err.(type) {
case nil:
    break
case *autoquery.ErrParsingComplete:
    // all query items have been parsed
    fmt.Println(err)
default:
    fmt.Println("unexpected error:", err)
}

Viability rules for index selection

In order for a given expression to be executed on a table, at least one index must meet all of the following criteria:

  • The partition key attribute of the index must be used in an Equal condition in the expression.
  • If an OrderBy attribute is specified on the expression, then the index must have the same attribute as its sort key.
  • If the expression contains a Select clause, then the index must include all selected attributes in its projection or project all attributes.
  • If the expression does not select attributes, then the index must project all attributes.
  • If the index is considered sparse (see below), then both the partition key and sort key attributes must appear in the expression. The sort key attribute may appear as the OrderBy clause for the index to be considered viable. It is not sufficient for the attribute to appear only in the Select clause.
  • If the expression specifies ConsistentRead(true), then the index must not be a global secondary index.

In general, autoquery should not be expected as a means of enabling full SQL-like flexibility. The expression capability still depends on the indexes defined for a table. Expression builds should be controlled in ways that guarantee there will be viable indexes for any expression build, such as requiring at least one attribute from a set of index partition keys.

Index Sparsity (advanced)

Sparse indexes do not contain all entries in the table. Consequently, indexes that are non-sparse are considered viable for a wider range of expressions as they only require the partition key attribute to appear in the expression as an Equal condition. However, sparse indexes may be preferred when they are viable as they will generally return all results while evaluating fewer items.

By default, the primary table index is considered non-sparse and all secondary indexes are considered sparse, with one exception*. This behavior can be modified at the client level by setting Client.SecondaryIndexSparsenessThreshold. If this threshold value is set to 1.0, then any secondary indexes whose size matches the total size of the table (at the time the metadata is queried) is considered non-sparse, and all other secondary indexes will be considered sparse, with one exception*. If this value is set between 0.0 and 1.0, then any index whose ratio of index items to total table items is greater than or equal to the threshold will be considered non-sparse, and all other secondary indexes will be considered sparse, with one exception*. If the value is set to 0.0 or less, then all indexes will be considered non-sparse.

*There is one exception to secondary index sparseness: since the primary table index attributes must be present in all items, any secondary index which uses either the table's primary partition key or primary sort key as its own sort key will always be non-sparse for purposes of index selection.

It is possible for the metadata and sparseness classification to become stale if items are added to the table which do not contain the secondary index's sort key attribute. If unsure about which indexes may be considered non-sparse, then it is recommended not to change SecondaryIndexSparsenessThreshold.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Client 

type Client struct {

	// SecondaryIndexSparsenessThreshold sets the threshold for secondary indexes to be considered
	// sparse vs non-sparse.
	//
	// A sparse index is only viable with expressions that include conditions for both the
	// partition key (which must be an Equal condition) and the sort key.
	//
	// The table's primary index is always non-sparse and is viable with any expression that
	// includes an Equal condition on the partition key.
	//
	// When a table's metadata is gathered, if the ratio of number of items in the secondary index
	// to number of items in the table is greater than or equal to
	// SecondaryIndexSparsenessThreshold, then the index will be considered non-sparse for
	// purposes of index selection.
	//
	// If the SecondaryIndexSparsenessThreshold is set to a value less than or equal to 0.0, then
	// all secondary indexes will be considered non-sparse; if set to 1.0, then each secondary
	// index will be considered non-sparse if the number of items in the index matches the total
	// number of items in the table. If set to a value greater than 1.0, then all secondary
	// indexes will be considered sparse. If the table is empty when sparseness is determined,
	// then every secondary index will be considered sparse unless the threshold is 0.0 or less.
	//
	// By default, all secondary indexes are considered sparse. If non-default behavior is
	// desired, this value should be set before any queries are parsed with Parser.Next.
	SecondaryIndexSparsenessThreshold float64
	// contains filtered or unexported fields
}

Client is a querying client for DynamoDB that enables automatic index selection. The client caches table metadata to optimize calls on previously-queried tables.

func NewClient 

func NewClient(service dynamodbiface.DynamoDBAPI) *Client

NewClient creates a new Client instance.

func NewClientWithMetadataProvider 

func NewClientWithMetadataProvider(
	service dynamodbiface.DynamoDBAPI, provider TableDescriptionProvider) *Client

NewClientWithMetadataProvider creates a new Client instance with a specified metadata provider.

Specifying alternate metadata providers is an advanced feature. Most users should use NewClient when creating an auto-querying client, which uses DescribeTable as the metadata provider.

An alternative TableDescriptionProvider may be needed in cases where the table cannot be described using DescribeTable.

func (*Client) Get 

func (client *Client) Get(ctx context.Context, tableName string, itemKey,
	returnItem interface{}) error

Get retrieves a single item by its key. The key is specified in itemKey and should be a struct with the appropriate dynamodbav attribute tags pertaining to the table's primary key. The item is returned in returnItem, which should have dynamodbav attribute tags pertaining to the desired return attributes in the table.

If the item is not found, an *ErrItemNotFound instance is returned.

func (*Client) Put 

func (client *Client) Put(ctx context.Context, tableName string, item interface{}) error

Put inserts a new item into the table, or replaces it if an item with the same primary key already exists. The item should be a struct with the appropriate dynamodbav attribute tags.

func (*Client) Query 

func (client *Client) Query(tableName string, expr *Expression) *Parser

Query initializes a query defined by expr on a table. The returned parser may be used to retrieve items using Parser.Next.

On the first call to a new table, the client will populate the table's index metadata using the underlying metadata provider. The metadata is cached for subsequent queries to the table through the same Client instance. The query automatically selects an index based on the table metadata and any expression restrictions.

func (*Client) Table 

func (client *Client) Table(tableName string) *Table

Table initializes a new Table instance from an autoquery client.

type ConditionKey 

type ConditionKey struct {
	// contains filtered or unexported fields
}

ConditionKey forms part of a condition of an expression.

The ConditionKey should be followed by a value condition in order to form a complete expression.

func Key 

func Key(attr string) *ConditionKey

Key begins a new expression with the key part of the condition.

The resulting ConditionKey should be followed by a condition in order to form a complete expression.

func (*ConditionKey) BeginsWith 

func (key *ConditionKey) BeginsWith(prefix string) *Expression

BeginsWith adds a new begins-with condition to the expression. Only items where the value of the key attribute begins with the specified prefix will be returned.

func (*ConditionKey) Between 

func (key *ConditionKey) Between(lowval, highval interface{}) *Expression

Between adds a new between condition to the expression. Only items where the value of the key attribute is between lowval and highval will be returned.

func (*ConditionKey) Equal 

func (key *ConditionKey) Equal(v interface{}) *Expression

Equal adds a new equal condition to the expression. Only items where the value of the key attribute equals v will be returned. All query expressions require at least one equal condition where the specified key attribute is an index partition key.

func (*ConditionKey) GreaterThan 

func (key *ConditionKey) GreaterThan(v interface{}) *Expression

GreaterThan adds a new greater than condition to the expression. Only items where the value of the key attribute is greater than v will be returned.

func (*ConditionKey) GreaterThanEqual 

func (key *ConditionKey) GreaterThanEqual(v interface{}) *Expression

GreaterThanEqual adds a new greater than or equal condition to the expression. Only items where the value of the key attribute is greater than or equal to v will be returned.

func (*ConditionKey) LessThan 

func (key *ConditionKey) LessThan(v interface{}) *Expression

LessThan adds a new less than condition to the expression. Only items where the value of the key attribute is less than v will be returned.

func (*ConditionKey) LessThanEqual 

func (key *ConditionKey) LessThanEqual(v interface{}) *Expression

LessThanEqual adds a new less than or equal condition to the expression. Only items where the value of the key attribute is less than or equal to v will be returned.

type ErrIndexNotViable 

type ErrIndexNotViable struct {
	IndexName        string   `json:"indexName"`
	NotViableReasons []string `json:"notViableReasons,omitempty"`
}

ErrIndexNotViable is returned when a specified index is not usable for the requested expression. The string returned by ErrIndexNotViable.Error includes reasons why the index is considered non-viable.

func (ErrIndexNotViable) Error 

func (e ErrIndexNotViable) Error() string

type ErrItemNotFound 

type ErrItemNotFound struct{}

ErrItemNotFound is returned by Get when an item with the provided key is not found in the table.

func (ErrItemNotFound) Error 

func (ErrItemNotFound) Error() string

type ErrNoViableIndexes 

type ErrNoViableIndexes struct {
	IndexErrs []*ErrIndexNotViable
}

ErrNoViableIndexes is returned by Client.Query when no table indexes are usable for the requested expression. The string returned by ErrNoViableIndexes.Error includes reasons why each index is considered non-viable.

func (ErrNoViableIndexes) Error 

func (e ErrNoViableIndexes) Error() string

type ErrParsingComplete 

type ErrParsingComplete struct {
	// contains filtered or unexported fields
}

ErrParsingComplete is returned by Parser.Next when all query items have been returned or when max pagination has been reached.

func (ErrParsingComplete) Error 

func (e ErrParsingComplete) Error() string

type Expression 

type Expression struct {
	// contains filtered or unexported fields
}

Expression contains conditions and filters to be used in a query.

func NewExpression 

func NewExpression() *Expression

NewExpression creates a new Expression instance.

func (*Expression) And 

func (expr *Expression) And(attr string) *ConditionKey

And begins a new condition on an existing expression.

The resulting ConditionKey should be followed by a condition in order to form a complete expression.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) BeginsWith 

func (expr *Expression) BeginsWith(attr string, prefix string) *Expression

BeginsWith adds a new begins-with condition to the expression. Only items where the value of the attribute attr begins with the specified prefix will be returned.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) Between 

func (expr *Expression) Between(attr string, lowval, highval interface{}) *Expression

Between adds a new between condition to the expression. Only items where the value of the attribute attr is between lowval and highval will be returned.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) ConsistentRead 

func (expr *Expression) ConsistentRead(val bool) *Expression

ConsistentRead sets the read consistency of each query page request. Note that consistent read only guarantees consistency within each page. Consistent read is not supported across all items in the query when pagination is required to parse all items (i.e. when the query evaluates more than 1MB of data). Consistent read is not supported on global secondary indexes.

func (*Expression) Equal 

func (expr *Expression) Equal(attr string, v interface{}) *Expression

Equal adds a new equal condition to the expression. Only items where the value of the attribute attr equals v will be returned. All query expressions require at least one equal condition where the specified attribute attr is an index partition key.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) Filter 

func (expr *Expression) Filter(filterCondition expression.ConditionBuilder) *Expression

Filter applies a condition from the DynamoDB expression package to an expression. Subsequent calls to Filter will append additional filters, and all filters will be applied as part of the expression.

These filters may be used to enable conditions that are otherwise not supported, such as OR conditions.

func (*Expression) GreaterThan 

func (expr *Expression) GreaterThan(attr string, v interface{}) *Expression

GreaterThan adds a new greater than condition to the expression. Only items where the value of the attribute attr is greater than v will be returned.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) GreaterThanEqual 

func (expr *Expression) GreaterThanEqual(attr string, v interface{}) *Expression

GreaterThanEqual adds a new greater than or equal condition to the expression. Only items where the value of the attribute attr is greater than or equal to v will be returned.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) LessThan 

func (expr *Expression) LessThan(attr string, v interface{}) *Expression

LessThan adds a new less than condition to the expression. Only items where the value of the attribute attr is less than v will be returned.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) LessThanEqual 

func (expr *Expression) LessThanEqual(attr string, v interface{}) *Expression

LessThanEqual adds a new less than or equal condition to the expression. Only items where the value of the attribute attr is less than or equal to v will be returned.

If multiple filter conditions are specified on the same attribute, only the most recent condition will apply to the expression.

func (*Expression) OrderBy 

func (expr *Expression) OrderBy(attr string, ascending bool) *Expression

OrderBy sets attr as the sort attribute. If ascending is true, items will be returned starting with the lowest value for the attribute. If ascending is false, the highest value will be returned first. OrderBy may only be used on sort key attributes of indexes which satisfy all other expression criteria.

func (*Expression) Select 

func (expr *Expression) Select(attrs ...string) *Expression

Select specifies attributes that should be returned in queried items. Subsequent calls to Select will append to the existing selected attributes for the expression.

If Select is not specified for an expression, the query will project all attributes for each returned item, but can only use indexes which project all attributes. When Select is specified, any indexes which include every selected attribute and satisfy all other expression criteria will be considered for the query index.

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

Parser is used for parsing query results.

func (*Parser) Next 

func (parser *Parser) Next(ctx context.Context, returnItem interface{}) error

Next retrieves the next item in the query. The returnItem is unmarshaled with "dynamodbav" struct tags.

On the first call to Next with a new table, the table's index metadata will be retrieved using the underlying metadata provider. For the default client created by NewClient, this requires IAM permissions to describe the table. The metadata is cached for subsequent queries to the table through the client instance used in the call to Query.

The first call to Next on a new Parser always makes a query call to DynamoDB. The query automatically selects an index based on the table metadata and any expression restrictions. On subsequent calls, the remaining buffered items will be returned in order until all buffered items have been returned. Next will make subsequent paginated query calls to DynamoDB to refill the internal buffer as necessary until max pages have been parsed completely or until all items in the query have been returned, whichever comes first. If no viable indexes are found, the call returns an ErrNoViableIndexes error.

Once all items have been returned or max pagination has been reached, the query will return ErrParsingComplete.

func (*Parser) SetExclusiveStartKey 

func (parser *Parser) SetExclusiveStartKey(
	exclusiveStartKey map[string]*dynamodb.AttributeValue) *Parser

SetExclusiveStartKey sets the exclusive start key for the next page query call to DynamoDB.

func (*Parser) SetLimitPerPage 

func (parser *Parser) SetLimitPerPage(limit int) *Parser

SetLimitPerPage sets the limit parameter for each page query call to DynamoDB. The limit parameter restricts the number of evaluated items, not the number of returned items.

func (*Parser) SetMaxPagination 

func (parser *Parser) SetMaxPagination(maxPages int) *Parser

SetMaxPagination sets the maximum number of pages to query. By default, the parser will consume additional pages until all query items have been read.

func (*Parser) UnsetLimitPerPage 

func (parser *Parser) UnsetLimitPerPage() *Parser

UnsetLimitPerPage unsets the limit parameter for each page query call to DynamoDB.

func (*Parser) UnsetMaxPagination 

func (parser *Parser) UnsetMaxPagination() *Parser

UnsetMaxPagination unsets the maximum pagination limit.

type Table 

type Table struct {
	// contains filtered or unexported fields
}

Table represents a specific DynamoDB table.

func (Table) Get 

func (table Table) Get(ctx context.Context, itemKey, returnItem interface{}) error

Get retrieves a single item by its key. The key is specified in itemKey and should be a struct with the appropriate dynamodbav attribute tags pertaining to the table's primary key. The item is returned in returnItem, which should have dynamodbav attribute tags pertaining to the desired return attributes in the table.

If the item is not found, ErrItemNotFound is returned.

func (Table) Put 

func (table Table) Put(ctx context.Context, item interface{}) error

Put inserts a new item into the table, or replaces it if an item with the same primary key already exists. The item should be a struct with the appropriate dynamodbav attribute tags.

func (Table) Query 

func (table Table) Query(expr *Expression) *Parser

Query initializes a query defined by expr on a table. The returned parser may be used to retrieve items using Parser.Next.

type TableDescriptionProvider 

type TableDescriptionProvider interface {
	Get(ctx context.Context, tableName string) (*dynamodb.TableDescription, error)
}

TableDescriptionProvider is used to gather DynamoDB table metadata.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.