physical

package
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 10, 2024 License: MPL-2.0 Imports: 22 Imported by: 90

Documentation

Index

Constants

View Source 
const (
	DeleteOperation Operation = "delete"
	GetOperation              = "get"
	ListOperation             = "list"
	PutOperation              = "put"
)
View Source 
const (
	ErrValueTooLarge = "put failed due to value being too large"
	ErrKeyTooLarge   = "put failed due to key being too large"
)
View Source 
const (
	// DefaultCacheSize is used if no cache size is specified for NewCache
	DefaultCacheSize = 128 * 1024
)
View Source 
const (
	// DefaultErrorPercent is used to determin how often we error
	DefaultErrorPercent = 20
)
View Source 
const (
	// DefaultJitterPercent is used if no cache size is specified for NewCache
	DefaultJitterPercent = 20
)
View Source 
const DefaultParallelOperations = 128

Variables

View Source 
var (
	ErrNonUTF8      = errors.New("key contains invalid UTF-8 characters")
	ErrNonPrintable = errors.New("key contains non-printable characters")
)
View Source 
var ErrRelativePath = errors.New("relative paths not supported")

Functions

func CacheRefreshContext added in v0.2.0 

func CacheRefreshContext(ctx context.Context, r bool) context.Context

CacheRefreshContext returns a context with an added value denoting if the cache should attempt a refresh.

func ExerciseBackend 

func ExerciseBackend(t testing.TB, b Backend)

func ExerciseBackend_ListPrefix 

func ExerciseBackend_ListPrefix(t testing.TB, b Backend)

func ExerciseHABackend 

func ExerciseHABackend(t testing.TB, b HABackend, b2 HABackend)

func ExerciseTransactionalBackend 

func ExerciseTransactionalBackend(t testing.TB, b Backend)

func GenericTransactionHandler 

func GenericTransactionHandler(ctx context.Context, t PseudoTransactional, txns []*TxnEntry) (retErr error)

Implements the transaction interface

func IsUnfencedWrite added in v0.10.1 

func IsUnfencedWrite(ctx context.Context) bool

IsUnfencedWrite returns whether or not the context passed has the unfenced flag value set.

func Prefixes 

func Prefixes(s string) []string

Prefixes is a shared helper function returns all parent 'folders' for a given vault key. e.g. for 'foo/bar/baz', it returns ['foo', 'foo/bar']

func UnfencedWriteCtx added in v0.10.1 

func UnfencedWriteCtx(ctx context.Context) context.Context

UnfencedWriteCtx adds metadata to a ctx such that any writes performed directly on a FencingHABackend using that context will _not_ add a fencing token.

Types

type Backend 

type Backend interface {
	// Put is used to insert or update an entry
	Put(ctx context.Context, entry *Entry) error

	// Get is used to fetch an entry
	Get(ctx context.Context, key string) (*Entry, error)

	// Delete is used to permanently delete an entry
	Delete(ctx context.Context, key string) error

	// List is used to list all the keys under a given
	// prefix, up to the next prefix.
	List(ctx context.Context, prefix string) ([]string, error)
}

Backend is the interface required for a physical backend. A physical backend is used to durably store data outside of Vault. As such, it is completely untrusted, and is only accessed via a security barrier. The backends must represent keys in a hierarchical manner. All methods are expected to be thread safe.

func NewStorageEncoding 

func NewStorageEncoding(b Backend) Backend

NewStorageEncoding returns a wrapped physical backend and verifies the key encoding

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is used to wrap an underlying physical backend and provide an LRU cache layer on top. Most of the reads done by Vault are for policy objects so there is a large read reduction by using a simple write-through cache.

func NewCache 

func NewCache(b Backend, size int, logger log.Logger, metricSink metrics.MetricSink) *Cache

NewCache returns a physical cache of the given size. If no size is provided, the default size is used.

func (*Cache) Delete 

func (c *Cache) Delete(ctx context.Context, key string) error

func (*Cache) Get 

func (c *Cache) Get(ctx context.Context, key string) (*Entry, error)

func (*Cache) List 

func (c *Cache) List(ctx context.Context, prefix string) ([]string, error)

func (*Cache) Purge 

func (c *Cache) Purge(ctx context.Context)

Purge is used to clear the cache

func (*Cache) Put 

func (c *Cache) Put(ctx context.Context, entry *Entry) error

func (*Cache) SetEnabled 

func (c *Cache) SetEnabled(enabled bool)

SetEnabled is used to toggle whether the cache is on or off. It must be called with true to actually activate the cache after creation.

func (*Cache) ShouldCache 

func (c *Cache) ShouldCache(key string) bool

type Entry 

type Entry struct {
	Key      string
	Value    []byte
	SealWrap bool `json:"seal_wrap,omitempty"`

	// Only used in replication
	ValueHash []byte
}

Entry is used to represent data stored by the physical backend

func (*Entry) String added in v0.6.0 

func (e *Entry) String() string

type ErrorInjector 

type ErrorInjector struct {
	// contains filtered or unexported fields
}

ErrorInjector is used to add errors into underlying physical requests

func NewErrorInjector 

func NewErrorInjector(b Backend, errorPercent int, logger log.Logger) *ErrorInjector

NewErrorInjector returns a wrapped physical backend to inject error

func (*ErrorInjector) Delete 

func (e *ErrorInjector) Delete(ctx context.Context, key string) error

func (*ErrorInjector) Get 

func (e *ErrorInjector) Get(ctx context.Context, key string) (*Entry, error)

func (*ErrorInjector) List 

func (e *ErrorInjector) List(ctx context.Context, prefix string) ([]string, error)

func (*ErrorInjector) Put 

func (e *ErrorInjector) Put(ctx context.Context, entry *Entry) error

func (*ErrorInjector) SetErrorPercentage 

func (e *ErrorInjector) SetErrorPercentage(p int)

type Factory 

type Factory func(config map[string]string, logger log.Logger) (Backend, error)

Factory is the factory function to create a physical backend.

type FencingHABackend added in v0.10.1 

type FencingHABackend interface {
	HABackend

	RegisterActiveNodeLock(l Lock) error
}

FencingHABackend is an HABackend which provides the additional guarantee that each Lock it returns from LockWith is also a FencingLock. A FencingLock provides a mechanism to retrieve a fencing token that can be included by future writes by the backend to ensure that it is still the current lock holder at the time the write commits. Without this timing might allow a lock holder not to notice it's no longer the active node for long enough for it to write data to storage even while a new active node is writing causing corruption. For Consul backend the fencing token is the session id which is submitted with `check-session` operation on each write to ensure the write only completes if the session is still holding the lock. For raft backend this isn't needed because our in-process raft library is unable to write if it's not the leader anyway.

If you implement this, Vault will call RegisterActiveNodeLock with the Lock instance returned by LockWith after it successfully locks it. This keeps the backend oblivious to the specific key we use for active node locks and allows potential future usage of locks for other purposes in the future.

Note that all implementations must support writing to storage before RegisterActiveNodeLock is called to support initialization of a new cluster. They must also skip fencing writes if the write's Context contains a special value. This is necessary to allow Vault to clear and re-initialise secondary clusters even though there is already an active node with a specific lock session since we clear the cluster while Vault is sealed and clearing the data might remove the lock in some storages (e.g. Consul). As noted above it's not generally safe to allow unfenced writes after a lock so instead we special case just a few types of writes that only happen rarely while the cluster is sealed. See the IsUnfencedWrite helper function.

type HABackend 

type HABackend interface {
	// LockWith is used for mutual exclusion based on the given key.
	LockWith(key, value string) (Lock, error)

	// Whether or not HA functionality is enabled
	HAEnabled() bool
}

HABackend is an extensions to the standard physical backend to support high-availability. Vault only expects to use mutual exclusion to allow multiple instances to act as a hot standby for a leader that services all requests.

type LatencyInjector 

type LatencyInjector struct {
	// contains filtered or unexported fields
}

LatencyInjector is used to add latency into underlying physical requests

func NewLatencyInjector 

func NewLatencyInjector(b Backend, latency time.Duration, jitter int, logger log.Logger) *LatencyInjector

NewLatencyInjector returns a wrapped physical backend to simulate latency

func (*LatencyInjector) Delete 

func (l *LatencyInjector) Delete(ctx context.Context, key string) error

Delete is a latent delete request

func (*LatencyInjector) Get 

func (l *LatencyInjector) Get(ctx context.Context, key string) (*Entry, error)

Get is a latent get request

func (*LatencyInjector) List 

func (l *LatencyInjector) List(ctx context.Context, prefix string) ([]string, error)

List is a latent list request

func (*LatencyInjector) Put 

func (l *LatencyInjector) Put(ctx context.Context, entry *Entry) error

Put is a latent put request

func (*LatencyInjector) SetLatency 

func (l *LatencyInjector) SetLatency(latency time.Duration)

type Lock 

type Lock interface {
	// Lock is used to acquire the given lock
	// The stopCh is optional and if closed should interrupt the lock
	// acquisition attempt. The return struct should be closed when
	// leadership is lost.
	Lock(stopCh <-chan struct{}) (<-chan struct{}, error)

	// Unlock is used to release the lock
	Unlock() error

	// Returns the value of the lock and if it is held by _any_ node
	Value() (bool, string, error)
}

type MountTableLimitingBackend added in v0.12.0 

type MountTableLimitingBackend interface {
	// RegisterMountTablePath informs the Backend that the given path represents
	// part of the mount tables or related metadata. This allows the backend to
	// apply different limits for this entry if configured to do so.
	RegisterMountTablePath(path string)
}

MountTableLimitingBackend is an optional interface a Backend can implement that allows it to support different entry size limits for mount-table-related paths. It will only be called in Vault Enterprise.

type Operation 

type Operation string

The operation type

type PermitPool 

type PermitPool struct {
	// contains filtered or unexported fields
}

PermitPool is used to limit maximum outstanding requests

func NewPermitPool 

func NewPermitPool(permits int) *PermitPool

NewPermitPool returns a new permit pool with the provided number of permits

func (*PermitPool) Acquire 

func (c *PermitPool) Acquire()

Acquire returns when a permit has been acquired

func (*PermitPool) CurrentPermits added in v0.2.0 

func (c *PermitPool) CurrentPermits() int

Get number of requests in the permit pool

func (*PermitPool) Release 

func (c *PermitPool) Release()

Release returns a permit to the pool

type PhysicalAccess 

type PhysicalAccess struct {
	// contains filtered or unexported fields
}

PhysicalAccess is a wrapper around physical.Backend that allows Core to expose its physical storage operations through PhysicalAccess() while restricting the ability to modify Core.physical itself.

func NewPhysicalAccess 

func NewPhysicalAccess(physical Backend) *PhysicalAccess

func (*PhysicalAccess) Delete 

func (p *PhysicalAccess) Delete(ctx context.Context, key string) error

func (*PhysicalAccess) Get 

func (p *PhysicalAccess) Get(ctx context.Context, key string) (*Entry, error)

func (*PhysicalAccess) List 

func (p *PhysicalAccess) List(ctx context.Context, prefix string) ([]string, error)

func (*PhysicalAccess) Purge 

func (p *PhysicalAccess) Purge(ctx context.Context)

func (*PhysicalAccess) Put 

func (p *PhysicalAccess) Put(ctx context.Context, entry *Entry) error

type PseudoTransactional 

type PseudoTransactional interface {
	// An internal function should do no locking or permit pool acquisition.
	// Depending on the backend and if it natively supports transactions, these
	// may simply chain to the normal backend functions.
	GetInternal(context.Context, string) (*Entry, error)
	PutInternal(context.Context, *Entry) error
	DeleteInternal(context.Context, string) error
}

type RedirectDetect 

type RedirectDetect interface {
	// DetectHostAddr is used to detect the host address
	DetectHostAddr() (string, error)
}

RedirectDetect is an optional interface that an HABackend can implement. If they do, a redirect address can be automatically detected.

type StorageEncoding 

type StorageEncoding struct {
	Backend
}

StorageEncoding is used to add errors into underlying physical requests

func (*StorageEncoding) Delete 

func (e *StorageEncoding) Delete(ctx context.Context, key string) error

func (*StorageEncoding) Purge 

func (e *StorageEncoding) Purge(ctx context.Context)

func (*StorageEncoding) Put 

func (e *StorageEncoding) Put(ctx context.Context, entry *Entry) error

func (*StorageEncoding) SetEnabled 

func (e *StorageEncoding) SetEnabled(enabled bool)

type TestTransactionalLimitBackend added in v0.11.0 

type TestTransactionalLimitBackend struct {
	TestTransactionalNonLimitBackend

	MaxEntries, MaxSize int
}

func (*TestTransactionalLimitBackend) TransactionLimits added in v0.11.0 

func (b *TestTransactionalLimitBackend) TransactionLimits() (int, int)

type TestTransactionalNonLimitBackend added in v0.11.0 

type TestTransactionalNonLimitBackend struct{}

func (*TestTransactionalNonLimitBackend) Delete added in v0.11.0 

func (b *TestTransactionalNonLimitBackend) Delete(ctx context.Context, key string) error

func (*TestTransactionalNonLimitBackend) Get added in v0.11.0 

func (b *TestTransactionalNonLimitBackend) Get(ctx context.Context, key string) (*Entry, error)

func (*TestTransactionalNonLimitBackend) List added in v0.11.0 

func (b *TestTransactionalNonLimitBackend) List(ctx context.Context, prefix string) ([]string, error)

func (*TestTransactionalNonLimitBackend) Put added in v0.11.0 

func (b *TestTransactionalNonLimitBackend) Put(ctx context.Context, entry *Entry) error

func (*TestTransactionalNonLimitBackend) Transaction added in v0.11.0 

func (b *TestTransactionalNonLimitBackend) Transaction(ctx context.Context, txns []*TxnEntry) error

type ToggleablePurgemonster 

type ToggleablePurgemonster interface {
	Purge(ctx context.Context)
	SetEnabled(bool)
}

ToggleablePurgemonster is an interface for backends that can toggle on or off special functionality and/or support purging. This is only used for the cache, don't use it for other things.

type Transactional 

type Transactional interface {
	// The function to run a transaction
	Transaction(context.Context, []*TxnEntry) error
}

Transactional is an optional interface for backends that support doing transactional updates of multiple keys. This is required for some features such as replication.

type TransactionalBackend 

type TransactionalBackend interface {
	Backend
	Transactional
}

type TransactionalCache 

type TransactionalCache struct {
	*Cache
	Transactional
}

TransactionalCache is a Cache that wraps the physical that is transactional

func NewTransactionalCache 

func NewTransactionalCache(b Backend, size int, logger log.Logger, metricSink metrics.MetricSink) *TransactionalCache

func (*TransactionalCache) LRU 

func (c *TransactionalCache) LRU() *lru.TwoQueueCache

func (*TransactionalCache) Locks 

func (c *TransactionalCache) Locks() []*locksutil.LockEntry

func (*TransactionalCache) Transaction 

func (c *TransactionalCache) Transaction(ctx context.Context, txns []*TxnEntry) error

func (*TransactionalCache) TransactionLimits added in v0.11.0 

func (c *TransactionalCache) TransactionLimits() (int, int)

TransactionLimits implements physical.TransactionalLimits

type TransactionalErrorInjector 

type TransactionalErrorInjector struct {
	*ErrorInjector
	Transactional
}

TransactionalErrorInjector is the transactional version of the error injector

func NewTransactionalErrorInjector 

func NewTransactionalErrorInjector(b Backend, errorPercent int, logger log.Logger) *TransactionalErrorInjector

NewTransactionalErrorInjector creates a new transactional ErrorInjector

func (*TransactionalErrorInjector) Transaction 

func (e *TransactionalErrorInjector) Transaction(ctx context.Context, txns []*TxnEntry) error

func (*TransactionalErrorInjector) TransactionLimits added in v0.11.0 

func (e *TransactionalErrorInjector) TransactionLimits() (int, int)

TransactionLimits implements physical.TransactionalLimits

type TransactionalLatencyInjector 

type TransactionalLatencyInjector struct {
	*LatencyInjector
	Transactional
}

TransactionalLatencyInjector is the transactional version of the latency injector

func NewTransactionalLatencyInjector 

func NewTransactionalLatencyInjector(b Backend, latency time.Duration, jitter int, logger log.Logger) *TransactionalLatencyInjector

NewTransactionalLatencyInjector creates a new transactional LatencyInjector jitter is the random percent that latency will vary between. For example, if you specify latency = 50ms and jitter = 20, then for any given operation, the latency will be 50ms +- 10ms (20% of 50), or between 40 and 60ms.

func (*TransactionalLatencyInjector) Transaction 

func (l *TransactionalLatencyInjector) Transaction(ctx context.Context, txns []*TxnEntry) error

Transaction is a latent transaction request

func (*TransactionalLatencyInjector) TransactionLimits added in v0.11.0 

func (l *TransactionalLatencyInjector) TransactionLimits() (int, int)

TransactionLimits implements physical.TransactionalLimits

type TransactionalLimits added in v0.11.0 

type TransactionalLimits interface {
	TransactionalBackend

	// TransactionLimits must return the limits of how large each transaction may
	// be. The limits returned indicate how many individual operation entries are
	// supported in total and an overall size limit on the contents of each
	// transaction if applicable. Vault will deduct any meta-operations it needs
	// to add from the maxEntries given. maxSize will be compared against the sum
	// of the key and value sizes for all operations in a transaction. The backend
	// should provide a reasonable margin of safety for any overhead it may have
	// while encoding, for example Consul's encoded transaction in JSON must fit
	// in the configured max transaction size so it must leave adequate room for
	// JSON encoding overhead on top of the raw key and value sizes.
	//
	// If zero is returned for either value, the replication internals will use
	// historic reasonable defaults. This allows middleware implementations such
	// as cache layers to either pass through to the underlying backend if it
	// implements this interface, or to return zeros to indicate that the
	// implementer should apply whatever defaults it would use if the middleware
	// were not present.
	TransactionLimits() (maxEntries int, maxSize int)
}

TransactionalLimits SHOULD be implemented by all TransactionalBackend implementations. It is separate for backwards compatibility reasons since this in a public SDK module. If a TransactionalBackend does not implement this, the historic default limits of 63 entries and 128kb (based on Consul's limits) are used by replication internals when encoding batches of transactions.

type TransactionalStorageEncoding 

type TransactionalStorageEncoding struct {
	*StorageEncoding
	Transactional
}

TransactionalStorageEncoding is the transactional version of the error injector

func (*TransactionalStorageEncoding) Transaction 

func (e *TransactionalStorageEncoding) Transaction(ctx context.Context, txns []*TxnEntry) error

func (*TransactionalStorageEncoding) TransactionLimits added in v0.11.0 

func (e *TransactionalStorageEncoding) TransactionLimits() (int, int)

TransactionLimits implements physical.TransactionalLimits

type TxnEntry 

type TxnEntry struct {
	Operation Operation
	Entry     *Entry
}

TxnEntry is an operation that takes atomically as part of a transactional update. Only supported by Transactional backends.

func SetupTestingTransactions 

func SetupTestingTransactions(t testing.TB, b Backend) []*TxnEntry

func (*TxnEntry) String added in v0.6.0 

func (t *TxnEntry) String() string

type View 

type View struct {
	// contains filtered or unexported fields
}

View represents a prefixed view of a physical backend

func NewView 

func NewView(backend Backend, prefix string) *View

NewView takes an underlying physical backend and returns a view of it that can only operate with the given prefix.

func (*View) Delete 

func (v *View) Delete(ctx context.Context, key string) error

Delete the entry from the prefix view

func (*View) Get 

func (v *View) Get(ctx context.Context, key string) (*Entry, error)

Get the key of the prefixed view

func (*View) List 

func (v *View) List(ctx context.Context, prefix string) ([]string, error)

List the contents of the prefixed view

func (*View) Put 

func (v *View) Put(ctx context.Context, entry *Entry) error

Put the entry into the prefix view

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.