blobstore

Documentation

Overview

Package blobstore provides a client interface for storing blobs behind orchestrator services.

Index

• Constants
• func GetBlobsByTypeAndKey(blobs []Blob) map[storage.TypeAndKey]Blob
• func GetTKsFromBlobs(blobs []Blob) []storage.TypeAndKey
• func GetTKsFromKeys(typ string, keys []string) []storage.TypeAndKey
• func ListKeysByNetwork(store TransactionalBlobStorage) (map[string][]storage.TypeAndKey, error)
• func P(networkID string, ids []storage.TypeAndKey) predicate.Blob
• type Blob
• type BlobStorageFactory
  • func NewEntStorage(tableName string, db *sql.DB, builder sqorc.StatementBuilder) BlobStorageFactory
  • func NewMemoryBlobStorageFactory() BlobStorageFactory
  • func NewSQLBlobStorageFactory(tableName string, db *sql.DB, sqlBuilder sqorc.StatementBuilder) BlobStorageFactory
• type LoadCriteria
  • func GetDefaultLoadCriteria() LoadCriteria
• type SearchFilter
  • func CreateSearchFilter(networkID *string, types []string, keys []string, keyPrefix *string) SearchFilter
  • func (sf SearchFilter) DoesTKMatch(tk storage.TypeAndKey) bool
  • func (sf SearchFilter) GetKeys() []string
  • func (sf SearchFilter) GetTypes() []string
• type TransactionalBlobStorage

Constants

const (
	CreateOrUpdate changeType = 1
	Delete         changeType = 2
)

Functions

func GetBlobsByTypeAndKey

func GetBlobsByTypeAndKey(blobs []Blob) map[storage.TypeAndKey]Blob

GetBlobsByTypeAndKey returns a computed view of a list of blobs as a map of blobs keyed by blob TypeAndKey.

func GetTKsFromBlobs

func GetTKsFromBlobs(blobs []Blob) []storage.TypeAndKey

GetTKsFromBlobs converts blobs to their associated type and key.

func GetTKsFromKeys

func GetTKsFromKeys(typ string, keys []string) []storage.TypeAndKey

GetTKsFromKeys returns the passed keys mapped as TypeAndKey, with the passed type applied to each.

func ListKeysByNetwork

func ListKeysByNetwork(store TransactionalBlobStorage) (map[string][]storage.TypeAndKey, error)

ListKeysByNetwork returns all blob keys, keyed by network ID.

func P

func P(networkID string, ids []storage.TypeAndKey) predicate.Blob

Types

type Blob

type Blob struct {
	Type    string
	Key     string
	Value   []byte
	Version uint64
}

Blob encapsulates a blob for storage.

type BlobStorageFactory

type BlobStorageFactory interface {
	InitializeFactory() error
	// StartTransaction opens a transaction for all following blob storage
	// operations, and returns a TransactionalBlobStorage instance tied to the
	// opened transaction.
	StartTransaction(opts *storage.TxOptions) (TransactionalBlobStorage, error)
}

BlobStorageFactory is an API to create a storage API bound to a transaction.

func NewEntStorage

func NewEntStorage(tableName string, db *sql.DB, builder sqorc.StatementBuilder) BlobStorageFactory

NewEntStorage returns an ent-based implementation of blobstore.

Note: due to constraints on how we use the ent-generated code across multiple tables, only one ent storage object can exist per process. As a result:

• DO NOT use more than one ent storage (table name) per service
• DO NOT use ent as the backing store for test services

func NewMemoryBlobStorageFactory

func NewMemoryBlobStorageFactory() BlobStorageFactory

NewMemoryBlobStorageFactory returns a BlobStorageFactory implementation which will return storage APIs backed by an in-memory map.

func NewSQLBlobStorageFactory

func NewSQLBlobStorageFactory(tableName string, db *sql.DB, sqlBuilder sqorc.StatementBuilder) BlobStorageFactory

NewSQLBlobStorageFactory returns a BlobStorageFactory implementation which will return storage APIs backed by SQL.

type LoadCriteria

type LoadCriteria struct {
	// LoadValue specifies whether to load the value of a blob.
	// Set to false to only load blob metadata.
	LoadValue bool
}

LoadCriteria specifies which fields of each blob should be loaded from the underlying store. Returned blobs will contain type-default values for non-loaded fields.

func GetDefaultLoadCriteria

func GetDefaultLoadCriteria() LoadCriteria

type SearchFilter

type SearchFilter struct {
	// Optional network ID to search within
	NetworkID *string

	// Limit search to an OR matching any of the specified types
	Types map[string]bool
	// Limit search to an OR matching any of the specified keys
	// If the KeyPrefix of the search filter is specified, this argument will
	// be ignored by the blobstore.
	Keys map[string]bool
	// Prefix to match keys against. If this is specified (non-nil and non-
	// empty), the values of Keys will be ignored.
	KeyPrefix *string
}

SearchFilter specifies search parameters. All fields are ANDed together in the final search that is performed.

func CreateSearchFilter

func CreateSearchFilter(networkID *string, types []string, keys []string, keyPrefix *string) SearchFilter

CreateSearchFilter creates a search filter for the given criteria. Nil elements result in no filtering. If you prefer to instantiate string sets manually, you can also create a SearchFilter directly.

func (SearchFilter) DoesTKMatch

func (sf SearchFilter) DoesTKMatch(tk storage.TypeAndKey) bool

DoesTKMatch returns true if the given TK matches the search filter, false otherwise.

func (SearchFilter) GetKeys

func (sf SearchFilter) GetKeys() []string

GetKeys returns the keys for this search filter sorted

func (SearchFilter) GetTypes

func (sf SearchFilter) GetTypes() []string

GetTypes returns the types for this search filter sorted

type TransactionalBlobStorage

type TransactionalBlobStorage interface {
	// Commit commits the existing transaction.
	// If an error is returned from the backing storage while committing,
	// the transaction will be rolled back.
	Commit() error

	// Rollback rolls back the existing transaction.
	// If the targeted transaction has already been committed,
	// rolling back has no effect and returns an error.
	Rollback() error

	// ListKeys returns all the blob keys stored for the network and type.
	ListKeys(networkID string, typeVal string) ([]string, error)

	// Get loads a specific blob from storage.
	// If there is no blob matching the given ID, ErrNotFound from
	// "github.com/go-magma/magma/lib/go/errors will be returned.
	Get(networkID string, id storage.TypeAndKey) (Blob, error)

	// GetMany loads and returns a collection of blobs matching the
	// specifiedIDs.
	// If there is no blob corresponding to a TypeAndKey, the returned list
	// will not have a corresponding Blob.
	GetMany(networkID string, ids []storage.TypeAndKey) ([]Blob, error)

	// Search returns a filtered collection of blobs keyed by the network ID
	// to which they belong.
	// Blobs are filtered according to the search filter. Empty filter returns
	// all blobs. Blobs contents are loaded according to the load criteria.
	// Empty criteria loads all fields.
	Search(filter SearchFilter, criteria LoadCriteria) (map[string][]Blob, error)

	// CreateOrUpdate writes blobs to the storage.
	// Blobs are either updated in-place or created. The Version field of
	// blobs passed here will be used if it is not set to 0, otherwise version
	// incrementation will be handled internally inside the storage
	// implementation.
	CreateOrUpdate(networkID string, blobs []Blob) error

	// GetExistingKeys takes in a list of keys and returns a list of keys that
	// exist from the input.
	// The filter specifies whether to look at the entire storage or just in
	// a network.
	GetExistingKeys(keys []string, filter SearchFilter) ([]string, error)

	// Delete deletes specified blobs from storage.
	Delete(networkID string, ids []storage.TypeAndKey) error

	// IncrementVersion is an atomic upsert (INSERT DO ON CONFLICT) that
	// increments the version column or inserts 1 if it does not exist.
	IncrementVersion(networkID string, id storage.TypeAndKey) error
}

TransactionalBlobStorage is the client API for blob storage operations within the context of a transaction. TODO(4/9/2020): refactor Get-like methods into package-level defaults wrapping Search -- see e.g. ListKeysByNetwork