backend

package
v1.2.3-fred.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 18, 2022 License: Apache-2.0 Imports: 23 Imported by: 405

Documentation

Overview

Package backend provides storage backend abstraction layer

backend package allows for pluggable back-ends for secrets storage. To implement a new storage back-end you have to supply an object which:

  • implements backend.Backend interface
  • implements backend.NewFunc function

Index

Constants

View Source 
const (
	// DefaultBufferCapacity is a default circular buffer size
	// used by backends to fan out events
	DefaultBufferCapacity = 1024
	// DefaultBacklogGracePeriod is the default amount of time
	// that the circular buffer will tolerate an event backlog
	// in one of its watchers.
	DefaultBacklogGracePeriod = time.Second * 30
	// DefaultPollStreamPeriod is a default event poll stream period
	DefaultPollStreamPeriod = time.Second
	// DefaultEventsTTL is a default events TTL period
	DefaultEventsTTL = 10 * time.Minute
	// DefaultRangeLimit is used to specify some very large limit when limit is not specified
	// explicitly to prevent OOM due to infinite loops or other issues along those lines.
	DefaultRangeLimit = 2_000_000
)
View Source 
const (
	Forever time.Duration = 0
)

Forever means that object TTL will not expire unless deleted

View Source 
const NoLimit = 0

NoLimit specifies no limits

View Source 
const Separator = '/'

Separator is used as a separator between key parts

Variables

This section is empty.

Functions

func EarliestExpiry 

func EarliestExpiry(times ...time.Time) time.Time

EarliestExpiry returns first of the otherwise returns empty

func ExactKey 

func ExactKey(parts ...string) []byte

ExactKey is like Key, except a Separator is appended to the result path of Key. This is to ensure range matching of a path will only math child paths and not other paths that have the resulting path as a prefix.

func Expiry 

func Expiry(clock clockwork.Clock, ttl time.Duration) time.Time

Expiry converts ttl to expiry time, if ttl is 0 returns empty time

func FlagKey 

func FlagKey(parts ...string) []byte

func GetPaginationKey 

func GetPaginationKey(r types.Resource) string

GetPaginationKey returns the pagination key given resource.

func IterateRange 

func IterateRange(ctx context.Context, bk Backend, startKey []byte, endKey []byte, limit int, fn func([]Item) (stop bool, err error)) error

IterateRange is a helper for stepping over a range

func Key 

func Key(parts ...string) []byte

Key joins parts into path separated by Separator, makes sure path always starts with Separator ("/")

func MaskKeyName 

func MaskKeyName(keyName string) []byte

MaskKeyName masks the given key name. e.g "123456789" -> "******789"

func NextPaginationKey 

func NextPaginationKey(r types.Resource) string

NextPaginationKey returns the next pagination key. For resources that have the HostID in their keys, the next key will also have the HostID part.

func RangeEnd 

func RangeEnd(key []byte) []byte

RangeEnd returns end of the range for given key.

func RunWhileLocked 

func RunWhileLocked(ctx context.Context, backend Backend, lockName string, ttl time.Duration, fn func(context.Context) error) error

RunWhileLocked allows you to run a function while a lock is held.

func TTL 

func TTL(clock clockwork.Clock, expires time.Time) time.Duration

TTL returns TTL in duration units, rounds up to one second

Types

type Backend 

type Backend interface {
	// Create creates item if it does not exist
	Create(ctx context.Context, i Item) (*Lease, error)

	// Put puts value into backend (creates if it does not
	// exists, updates it otherwise)
	Put(ctx context.Context, i Item) (*Lease, error)

	// CompareAndSwap compares item with existing item
	// and replaces is with replaceWith item
	CompareAndSwap(ctx context.Context, expected Item, replaceWith Item) (*Lease, error)

	// Update updates value in the backend
	Update(ctx context.Context, i Item) (*Lease, error)

	// Get returns a single item or not found error
	Get(ctx context.Context, key []byte) (*Item, error)

	// GetRange returns query range
	GetRange(ctx context.Context, startKey []byte, endKey []byte, limit int) (*GetResult, error)

	// Delete deletes item by key, returns NotFound error
	// if item does not exist
	Delete(ctx context.Context, key []byte) error

	// DeleteRange deletes range of items with keys between startKey and endKey
	DeleteRange(ctx context.Context, startKey, endKey []byte) error

	// KeepAlive keeps object from expiring, updates lease on the existing object,
	// expires contains the new expiry to set on the lease,
	// some backends may ignore expires based on the implementation
	// in case if the lease managed server side
	KeepAlive(ctx context.Context, lease Lease, expires time.Time) error

	// NewWatcher returns a new event watcher
	NewWatcher(ctx context.Context, watch Watch) (Watcher, error)

	// Close closes backend and all associated resources
	Close() error

	// Clock returns clock used by this backend
	Clock() clockwork.Clock

	// CloseWatchers closes all the watchers
	// without closing the backend
	CloseWatchers()
}

Backend implements abstraction over local or remote storage backend. Item keys are assumed to be valid UTF8, which may be enforced by the various Backend implementations.

type Batch 

type Batch interface {
	// PutRange puts range of items in one transaction
	PutRange(ctx context.Context, items []Item) error
}

Batch implements some batch methods that are not mandatory for all interfaces, only the ones used in bulk operations.

type BufferOption 

type BufferOption func(*bufferConfig)

func BacklogGracePeriod 

func BacklogGracePeriod(d time.Duration) BufferOption

BacklogGracePeriod sets the amount of time a watcher with a backlog will be tolerated.

func BufferCapacity 

func BufferCapacity(c int) BufferOption

BufferCapacity sets the event capacity of the circular buffer.

func BufferClock 

func BufferClock(c clockwork.Clock) BufferOption

BufferClock sets a custom clock for the buffer (used in tests).

type BufferWatcher 

type BufferWatcher struct {
	Watch
	// contains filtered or unexported fields
}

BufferWatcher is a watcher connected to the buffer and receiving fan-out events from the watcher

func (*BufferWatcher) Close 

func (w *BufferWatcher) Close() error

Close closes the watcher, could be called multiple times, removes the watcher from the buffer queue

func (*BufferWatcher) Done 

func (w *BufferWatcher) Done() <-chan struct{}

Done channel is closed when watcher is closed

func (*BufferWatcher) Events 

func (w *BufferWatcher) Events() <-chan Event

Events returns events channel. This method performs internal work and should be re-called after each event is received, rather than having its output cached.

func (*BufferWatcher) String 

func (w *BufferWatcher) String() string

String returns user-friendly representation of the buffer watcher

type CircularBuffer 

type CircularBuffer struct {
	sync.Mutex
	*log.Entry
	// contains filtered or unexported fields
}

CircularBuffer implements in-memory circular buffer of predefined size, that is capable of fan-out of the backend events.

func NewCircularBuffer 

func NewCircularBuffer(opts ...BufferOption) *CircularBuffer

NewCircularBuffer returns a new uninitialized instance of circular buffer.

func (*CircularBuffer) Clear 

func (c *CircularBuffer) Clear()

Clear clears all events from the queue and closes all active watchers, but does not modify init state.

func (*CircularBuffer) Close 

func (c *CircularBuffer) Close() error

Close closes circular buffer and all watchers

func (*CircularBuffer) Emit 

func (c *CircularBuffer) Emit(events ...Event) (ok bool)

Emit emits events to currently registered watchers and stores them to the buffer. Panics if called before SetInit(), and returns false if called after Close().

func (*CircularBuffer) NewWatcher 

func (c *CircularBuffer) NewWatcher(ctx context.Context, watch Watch) (Watcher, error)

NewWatcher adds a new watcher to the events buffer

func (*CircularBuffer) Reset 

func (c *CircularBuffer) Reset()

Reset is equivalent to Clear except that is also sets the buffer into an uninitialized state. This method should only be used when resetting after a broken event stream. If only closure of watchers is desired, use Clear instead.

func (*CircularBuffer) SetInit 

func (c *CircularBuffer) SetInit()

SetInit puts the buffer into an initialized state if it isn't already. Any watchers already queued will be sent init events, and watchers added after this call will have their init events sent immediately. This function must be called *after* establishing a healthy parent event stream in order to preserve correct cache behavior.

type Config added in v1.2.6 

type Config struct {
	// Type can be "bolt" or "etcd" or "dynamodb"
	Type string `yaml:"type,omitempty"`

	// Params is a generic key/value property bag which allows arbitrary
	// values to be passed to backend
	Params Params `yaml:",inline"`
}

Config is used for 'storage' config section. It's a combination of values for various backends: 'boltdb', 'etcd', 'filesystem' and 'dynamodb'

type Event 

type Event struct {
	// Type is operation type
	Type types.OpType
	// Item is event Item
	Item Item
}

Event is a event containing operation with item

func (Event) String 

func (e Event) String() string

type GetResult 

type GetResult struct {
	// Items returns a list of items
	Items []Item
}

GetResult provides the result of GetRange request

type Item 

type Item struct {
	// Key is a key of the key value item
	Key []byte
	// Value is a value of the key value item
	Value []byte
	// Expires is an optional record expiry time
	Expires time.Time
	// ID is a record ID, newer records have newer ids
	ID int64
	// LeaseID is a lease ID, could be set on objects
	// with TTL
	LeaseID int64
}

Item is a key value item

type Items 

type Items []Item

Items is a sortable list of backend items

func (Items) Len 

func (it Items) Len() int

Len is part of sort.Interface.

func (Items) Less 

func (it Items) Less(i, j int) bool

Less is part of sort.Interface.

func (Items) Swap 

func (it Items) Swap(i, j int)

Swap is part of sort.Interface.

type Lease 

type Lease struct {
	// Key is an object representing lease
	Key []byte
	// ID is a lease ID, could be empty
	ID int64
}

Lease represents a lease on the item that can be used to extend item's TTL without updating its contents.

Here is an example of renewing object TTL:

item.Expires = time.Now().Add(10 * time.Second) lease, err := backend.Create(ctx, item) expires := time.Now().Add(20 * time.Second) err = backend.KeepAlive(ctx, lease, expires)

func (*Lease) IsEmpty 

func (l *Lease) IsEmpty() bool

IsEmpty returns true if the lease is empty value

type Lock 

type Lock struct {
	// contains filtered or unexported fields
}

func AcquireLock 

func AcquireLock(ctx context.Context, backend Backend, lockName string, ttl time.Duration) (Lock, error)

AcquireLock grabs a lock that will be released automatically in TTL

func (*Lock) Release 

func (l *Lock) Release(ctx context.Context, backend Backend) error

Release forces lock release

type Params 

type Params map[string]interface{}

Params type defines a flexible unified back-end configuration API. It is just a map of key/value pairs which gets populated by `storage` section in Teleport YAML config.

func (Params) GetString 

func (p Params) GetString(key string) string

GetString returns a string value stored in Params map, or an empty string if nothing is found

type Reporter 

type Reporter struct {
	// ReporterConfig contains reporter wrapper configuration
	ReporterConfig
	// contains filtered or unexported fields
}

Reporter wraps a Backend implementation and reports statistics about the backend operations

func NewReporter 

func NewReporter(cfg ReporterConfig) (*Reporter, error)

NewReporter returns a new Reporter.

func (*Reporter) Clock 

func (s *Reporter) Clock() clockwork.Clock

Clock returns clock used by this backend

func (*Reporter) Close 

func (s *Reporter) Close() error

Close releases the resources taken up by this backend

func (*Reporter) CloseWatchers 

func (s *Reporter) CloseWatchers()

CloseWatchers closes all the watchers without closing the backend

func (*Reporter) CompareAndSwap 

func (s *Reporter) CompareAndSwap(ctx context.Context, expected Item, replaceWith Item) (*Lease, error)

CompareAndSwap compares item with existing item and replaces is with replaceWith item

func (*Reporter) Create 

func (s *Reporter) Create(ctx context.Context, i Item) (*Lease, error)

Create creates item if it does not exist

func (*Reporter) Delete 

func (s *Reporter) Delete(ctx context.Context, key []byte) error

Delete deletes item by key

func (*Reporter) DeleteRange 

func (s *Reporter) DeleteRange(ctx context.Context, startKey []byte, endKey []byte) error

DeleteRange deletes range of items

func (*Reporter) Get 

func (s *Reporter) Get(ctx context.Context, key []byte) (*Item, error)

Get returns a single item or not found error

func (*Reporter) GetRange 

func (s *Reporter) GetRange(ctx context.Context, startKey []byte, endKey []byte, limit int) (*GetResult, error)

GetRange returns query range

func (*Reporter) KeepAlive 

func (s *Reporter) KeepAlive(ctx context.Context, lease Lease, expires time.Time) error

KeepAlive keeps object from expiring, updates lease on the existing object, expires contains the new expiry to set on the lease, some backends may ignore expires based on the implementation in case if the lease managed server side

func (*Reporter) NewWatcher 

func (s *Reporter) NewWatcher(ctx context.Context, watch Watch) (Watcher, error)

NewWatcher returns a new event watcher

func (*Reporter) Put 

func (s *Reporter) Put(ctx context.Context, i Item) (*Lease, error)

Put puts value into backend (creates if it does not exists, updates it otherwise)

func (*Reporter) Update 

func (s *Reporter) Update(ctx context.Context, i Item) (*Lease, error)

Update updates value in the backend

type ReporterConfig 

type ReporterConfig struct {
	// Backend is a backend to wrap
	Backend Backend
	// Component is a component name to report
	Component string
	// Number of the most recent backend requests to preserve for top requests
	// metric. Higher value means higher memory usage but fewer infrequent
	// requests forgotten.
	TopRequestsCount int
	// Tracer is used to create spans
	Tracer oteltrace.Tracer
}

ReporterConfig configures reporter wrapper

func (*ReporterConfig) CheckAndSetDefaults 

func (r *ReporterConfig) CheckAndSetDefaults() error

CheckAndSetDefaults checks and sets

type ReporterWatcher 

type ReporterWatcher struct {
	Watcher
	Component string
}

ReporterWatcher is a wrapper around backend watcher that reports events

func NewReporterWatcher 

func NewReporterWatcher(ctx context.Context, component string, w Watcher) *ReporterWatcher

NewReporterWatcher creates new reporter watcher instance

type Sanitizer 

type Sanitizer struct {
	// contains filtered or unexported fields
}

Sanitizer wraps a Backend implementation to make sure all values requested of the backend are whitelisted.

func NewSanitizer 

func NewSanitizer(backend Backend) *Sanitizer

NewSanitizer returns a new Sanitizer.

func (*Sanitizer) Clock 

func (s *Sanitizer) Clock() clockwork.Clock

Clock returns clock used by this backend

func (*Sanitizer) Close 

func (s *Sanitizer) Close() error

Close releases the resources taken up by this backend

func (*Sanitizer) CloseWatchers 

func (s *Sanitizer) CloseWatchers()

CloseWatchers closes all the watchers without closing the backend

func (*Sanitizer) CompareAndSwap 

func (s *Sanitizer) CompareAndSwap(ctx context.Context, expected Item, replaceWith Item) (*Lease, error)

CompareAndSwap compares item with existing item and replaces is with replaceWith item

func (*Sanitizer) Create 

func (s *Sanitizer) Create(ctx context.Context, i Item) (*Lease, error)

Create creates item if it does not exist

func (*Sanitizer) Delete 

func (s *Sanitizer) Delete(ctx context.Context, key []byte) error

Delete deletes item by key

func (*Sanitizer) DeleteRange 

func (s *Sanitizer) DeleteRange(ctx context.Context, startKey []byte, endKey []byte) error

DeleteRange deletes range of items

func (*Sanitizer) Get 

func (s *Sanitizer) Get(ctx context.Context, key []byte) (*Item, error)

Get returns a single item or not found error

func (*Sanitizer) GetRange 

func (s *Sanitizer) GetRange(ctx context.Context, startKey []byte, endKey []byte, limit int) (*GetResult, error)

GetRange returns query range

func (*Sanitizer) KeepAlive 

func (s *Sanitizer) KeepAlive(ctx context.Context, lease Lease, expires time.Time) error

KeepAlive keeps object from expiring, updates lease on the existing object, expires contains the new expiry to set on the lease, some backends may ignore expires based on the implementation in case if the lease managed server side

func (*Sanitizer) NewWatcher 

func (s *Sanitizer) NewWatcher(ctx context.Context, watch Watch) (Watcher, error)

NewWatcher returns a new event watcher

func (*Sanitizer) Put 

func (s *Sanitizer) Put(ctx context.Context, i Item) (*Lease, error)

Put puts value into backend (creates if it does not exists, updates it otherwise)

func (*Sanitizer) Update 

func (s *Sanitizer) Update(ctx context.Context, i Item) (*Lease, error)

Update updates value in the backend

type Watch 

type Watch struct {
	// Name is a watch name set for debugging
	// purposes
	Name string
	// Prefixes specifies prefixes to watch,
	// passed to the backend implementation
	Prefixes [][]byte
	// QueueSize is an optional queue size
	QueueSize int
	// MetricComponent if set will start reporting
	// with a given component metric
	MetricComponent string
}

Watch specifies watcher parameters

func (*Watch) String 

func (w *Watch) String() string

String returns a user-friendly description of the watcher

type Watcher 

type Watcher interface {
	// Events returns channel with events
	Events() <-chan Event

	// Done returns the channel signaling the closure
	Done() <-chan struct{}

	// Close closes the watcher and releases
	// all associated resources
	Close() error
}

Watcher returns watcher

type Wrapper 

type Wrapper struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Wrapper wraps a Backend implementation that can fail on demand.

func NewWrapper 

func NewWrapper(backend Backend) *Wrapper

NewWrapper returns a new Wrapper.

func (*Wrapper) Clock 

func (s *Wrapper) Clock() clockwork.Clock

Clock returns clock used by this backend

func (*Wrapper) Close 

func (s *Wrapper) Close() error

Close releases the resources taken up by this backend

func (*Wrapper) CloseWatchers 

func (s *Wrapper) CloseWatchers()

CloseWatchers closes all the watchers without closing the backend

func (*Wrapper) CompareAndSwap 

func (s *Wrapper) CompareAndSwap(ctx context.Context, expected Item, replaceWith Item) (*Lease, error)

CompareAndSwap compares item with existing item and replaces is with replaceWith item

func (*Wrapper) Create 

func (s *Wrapper) Create(ctx context.Context, i Item) (*Lease, error)

Create creates item if it does not exist

func (*Wrapper) Delete 

func (s *Wrapper) Delete(ctx context.Context, key []byte) error

Delete deletes item by key

func (*Wrapper) DeleteRange 

func (s *Wrapper) DeleteRange(ctx context.Context, startKey []byte, endKey []byte) error

DeleteRange deletes range of items

func (*Wrapper) Get 

func (s *Wrapper) Get(ctx context.Context, key []byte) (*Item, error)

Get returns a single item or not found error

func (*Wrapper) GetRange 

func (s *Wrapper) GetRange(ctx context.Context, startKey []byte, endKey []byte, limit int) (*GetResult, error)

GetRange returns query range

func (*Wrapper) GetReadError 

func (s *Wrapper) GetReadError() error

GetReadError returns error to be returned by read backend operations

func (*Wrapper) KeepAlive 

func (s *Wrapper) KeepAlive(ctx context.Context, lease Lease, expires time.Time) error

KeepAlive keeps object from expiring, updates lease on the existing object, expires contains the new expiry to set on the lease, some backends may ignore expires based on the implementation in case if the lease managed server side

func (*Wrapper) NewWatcher 

func (s *Wrapper) NewWatcher(ctx context.Context, watch Watch) (Watcher, error)

NewWatcher returns a new event watcher

func (*Wrapper) Put 

func (s *Wrapper) Put(ctx context.Context, i Item) (*Lease, error)

Put puts value into backend (creates if it does not exists, updates it otherwise)

func (*Wrapper) SetReadError 

func (s *Wrapper) SetReadError(err error)

SetReadError sets error to be returned by read backend operations

func (*Wrapper) Update 

func (s *Wrapper) Update(ctx context.Context, i Item) (*Lease, error)

Update updates value in the backend

Source Files

View all Source files

Directories

Path Synopsis
Package dynamo implements DynamoDB storage backend for Teleport auth service, similar to etcd backend.
 Package dynamo implements DynamoDB storage backend for Teleport auth service, similar to etcd backend.
Package etcdbk implements Etcd powered backend
 Package etcdbk implements Etcd powered backend
Package firestoreFirestoreBackend implements Firestore storage backend for Teleport auth service, similar to DynamoDB backend.
 Package firestoreFirestoreBackend implements Firestore storage backend for Teleport auth service, similar to DynamoDB backend.
Package kubernetes implements Kubernetes Secret backend used for persisting identity and state for agent's running in Kubernetes clusters.
 Package kubernetes implements Kubernetes Secret backend used for persisting identity and state for agent's running in Kubernetes clusters.
Package lite implements SQLite backend used for local persistent caches in proxies and nodes and for standalone auth service deployments.
 Package lite implements SQLite backend used for local persistent caches in proxies and nodes and for standalone auth service deployments.
Package memory implements backend interface using a combination of Minheap (to store expiring items) and B-Tree for storing sorted dictionary of items.
 Package memory implements backend interface using a combination of Minheap (to store expiring items) and B-Tree for storing sorted dictionary of items.
Package test contains a backend acceptance test suite that is backend implementation independent each backend will use the suite to test itself
 Package test contains a backend acceptance test suite that is backend implementation independent each backend will use the suite to test itself

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.