statemgr

package
v0.0.0-...-c924774 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 29, 2024 License: MPL-2.0 Imports: 30 Imported by: 0

Documentation

Overview

Package statemgr defines the interfaces and some supporting functionality for "state managers", which are components responsible for writing state to some persistent storage and then later retrieving it.

State managers will usually (but not necessarily) use the state file formats implemented in the sibling directory "statefile" to serialize the persistent parts of state for storage.

State managers are responsible for ensuring that stored state can be updated safely across multiple, possibly-concurrent OpenTofu runs (with reasonable constraints and limitations). The rest of OpenTofu considers state to be a mutable data structure, with state managers preserving that illusion by creating snapshots of the state and updating them over time.

From the perspective of callers of the general state manager API, a state manager is able to return the latest snapshot and to replace that snapshot with a new one. Some state managers may also preserve historical snapshots using facilities offered by their storage backend, but this is always an implementation detail: the historical versions are not visible to a user of these interfaces.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckValidImport 

func CheckValidImport(newFile, existingFile *statefile.File) error

CheckValidImport returns nil if the "new" snapshot can be imported as a successor of the "existing" snapshot without forcing.

If not, an error is returned describing why.

func Export 

func Export(mgr Reader) *statefile.File

Export retrieves the latest state snapshot from the given manager, including its metadata (serial and lineage) where possible.

A state manager must also implement either Migrator or PersistentMeta for the metadata to be included. Otherwise, the relevant fields will have zero value in the returned object.

For state managers that also implement Persistent, it is the caller's responsibility to refresh from persistent storage first if needed.

This function doesn't do any locking of its own, so if the state manager also implements Locker the caller should hold a lock on it for the duration of this call.

func Import 

func Import(f *statefile.File, mgr Transient, force bool) error

Import loads the given state snapshot into the given manager, preserving its metadata (serial and lineage) if the target manager supports metadata.

A state manager must implement the optional interface Migrator to get access to the full metadata.

Unless "force" is true, Import will check first that the metadata given in the file matches the current snapshot metadata for the manager, if the manager supports metadata. Some managers do not support forcing, so a write with an unsuitable lineage or serial may still be rejected even if "force" is set. "force" has no effect for managers that do not support snapshot metadata.

For state managers that also implement Persistent, it is the caller's responsibility to persist the newly-written state after a successful result, just as with calls to Writer.WriteState.

This function doesn't do any locking of its own, so if the state manager also implements Locker the caller should hold a lock on it for the duration of this call.

func LockWithContext 

func LockWithContext(ctx context.Context, s Locker, info *LockInfo) (string, error)

LockWithContext locks the given state manager using the provided context for both timeout and cancellation.

This method has a built-in retry/backoff behavior up to the context's timeout.

func Migrate 

func Migrate(dst, src Transient) error

Migrate writes the latest transient state snapshot from src into dest, preserving snapshot metadata (serial and lineage) where possible.

If both managers implement the optional interface Migrator then it will be used to copy the snapshot and its associated metadata. Otherwise, the normal Reader and Writer interfaces will be used instead.

If the destination manager refuses the new state or fails to write it then its error is returned directly.

For state managers that also implement Persistent, it is the caller's responsibility to persist the newly-written state after a successful result, just as with calls to Writer.WriteState.

This function doesn't do any locking of its own, so if the state managers also implement Locker the caller should hold a lock on both managers for the duration of this call.

func NewLineage 

func NewLineage() string

NewLineage generates a new lineage identifier string. A lineage identifier is an opaque string that is intended to be unique in space and time, chosen when state is recorded at a location for the first time and then preserved afterwards to allow OpenTofu to recognize when one state snapshot is a predecessor or successor of another.

func NewStateFile 

func NewStateFile() *statefile.File

NewStateFile creates a new statefile.File object, with a newly-minted lineage identifier and serial 0, and returns a pointer to it.

func PlannedStateUpdate 

func PlannedStateUpdate(mgr Transient, planned *states.State) *statefile.File

PlannedStateUpdate is a special helper to obtain a statefile representation of a not-yet-written state snapshot that can be written later by a call to the companion function WritePlannedStateUpdate.

The statefile object returned here has an unusual interpretation of its metadata that is understood only by WritePlannedStateUpdate, and so the returned object should not be used for any other purpose.

If the state manager implements Locker then it is the caller's responsibility to hold the lock at least for the duration of this call. It is not safe to modify the given state concurrently while PlannedStateUpdate is running.

func RefreshAndRead 

func RefreshAndRead(mgr Storage) (*states.State, error)

RefreshAndRead refreshes the persistent snapshot in the given state manager and then returns it.

This is a wrapper around calling RefreshState and then State on the given manager.

func TestFull 

func TestFull(t *testing.T, s Full)

TestFull is a helper for testing full state manager implementations. It expects that the given implementation is pre-loaded with a snapshot of the result from TestFullInitialState.

If the given state manager also implements PersistentMeta, this function will test that the snapshot metadata changes as expected between calls to the methods of Persistent.

func TestFullInitialState 

func TestFullInitialState() *states.State

TestFullInitialState is a state that should be snapshotted into a full state manager before passing it into TestFull.

func WriteAndPersist 

func WriteAndPersist(mgr Storage, state *states.State, schemas *tofu.Schemas) error

WriteAndPersist writes a snapshot of the given state to the given state manager's transient store and then immediately persists it.

The caller must ensure that the given state is not concurrently modified while this function is running, but it is safe to modify it after this function has returned.

If an error is returned, it is undefined whether the state has been saved to the transient store or not, and so the only safe response is to bail out quickly with a user-facing error. In situations where more control is required, call WriteState and PersistState on the state manager directly and handle their errors.

func WritePlannedStateUpdate 

func WritePlannedStateUpdate(mgr Transient, planned *statefile.File) error

WritePlannedStateUpdate is a companion to PlannedStateUpdate that attempts to apply a state update that was planned earlier to the given state manager.

An error is returned if this function detects that a new state snapshot has been written to the backend since the update was planned, since that invalidates the plan. An error is returned also if the manager itself rejects the given state when asked to store it.

If the returned error is nil, the given manager's transient state snapshot is updated to match what was planned. It is the caller's responsibility to then persist that state if the manager also implements Persistent and the snapshot should be written to the persistent store.

If the state manager implements Locker then it is the caller's responsibility to hold the lock at least for the duration of this call.

Types

type Filesystem 

type Filesystem struct {
	// contains filtered or unexported fields
}

Filesystem is a full state manager that uses a file in the local filesystem for persistent storage.

The transient storage for Filesystem is always in-memory.

func NewFilesystem 

func NewFilesystem(statePath string, enc encryption.StateEncryption) *Filesystem

NewFilesystem creates a filesystem-based state manager that reads and writes state snapshots at the given filesystem path.

This is equivalent to calling NewFileSystemBetweenPaths with statePath as both of the path arguments.

func NewFilesystemBetweenPaths 

func NewFilesystemBetweenPaths(readPath, writePath string, enc encryption.StateEncryption) *Filesystem

NewFilesystemBetweenPaths creates a filesystem-based state manager that reads an initial snapshot from readPath and then writes all new snapshots to writePath.

func (*Filesystem) BackupPath 

func (s *Filesystem) BackupPath() string

BackupPath returns the manager's backup path if backup files are enabled, or an empty string otherwise.

func (*Filesystem) GetRootOutputValues 

func (s *Filesystem) GetRootOutputValues() (map[string]*states.OutputValue, error)

func (*Filesystem) Lock 

func (s *Filesystem) Lock(info *LockInfo) (string, error)

Lock implements Locker using filesystem discretionary locks.

func (*Filesystem) PersistState 

func (s *Filesystem) PersistState(schemas *tofu.Schemas) error

PersistState writes state to a tfstate file.

func (*Filesystem) RefreshState 

func (s *Filesystem) RefreshState() error

RefreshState is an implementation of Refresher.

func (*Filesystem) SetBackupPath 

func (s *Filesystem) SetBackupPath(path string)

SetBackupPath configures the receiever so that it will create a local backup file of the next state snapshot it reads (in State) if a different snapshot is subsequently written (in WriteState). Only one backup is written for the lifetime of the object, unless reset as described below.

For correct operation, this must be called before any other state methods are called. If called multiple times, each call resets the backup function so that the next read will become the backup snapshot and a following write will save a backup of it.

func (*Filesystem) State 

func (s *Filesystem) State() *states.State

State is an implementation of Reader.

func (*Filesystem) StateForMigration 

func (s *Filesystem) StateForMigration() *statefile.File

StateForMigration is part of our implementation of Migrator.

func (*Filesystem) StateSnapshotMeta 

func (s *Filesystem) StateSnapshotMeta() SnapshotMeta

StateSnapshotMeta returns the metadata from the most recently persisted or refreshed persistent state snapshot.

This is an implementation of PersistentMeta.

func (*Filesystem) Unlock 

func (s *Filesystem) Unlock(id string) error

Unlock is the companion to Lock, completing the implemention of Locker.

func (*Filesystem) WriteState 

func (s *Filesystem) WriteState(state *states.State) error

WriteState is an implementation of Writer.

func (*Filesystem) WriteStateForMigration 

func (s *Filesystem) WriteStateForMigration(f *statefile.File, force bool) error

WriteStateForMigration is part of our implementation of Migrator.

type Full 

type Full interface {
	Storage
	Locker
}

Full is the union of all of the more-specific state interfaces.

This interface may grow over time, so state implementations aiming to implement it may need to be modified for future changes. To ensure that this need can be detected, always include a statement nearby the declaration of the implementing type that will fail at compile time if the interface isn't satisfied, such as: 

var _ statemgr.Full = (*ImplementingType)(nil)

func NewFullFake 

func NewFullFake(t Transient, initial *states.State) Full

NewFullFake returns a full state manager that really only supports transient snapshots. This is primarily intended for testing and is not suitable for general use.

The persistent part of the interface is stubbed out as an in-memory store, and so its snapshots are effectively also transient.

The given Transient implementation is used to implement the transient portion of the interface. If nil is given, NewTransientInMemory is automatically called to create an in-memory transient manager with no initial transient snapshot.

If the given initial state is non-nil then a copy of it will be used as the initial persistent snapshot.

The Locker portion of the returned manager uses a local mutex to simulate mutually-exclusive access to the fake persistent portion of the object.

func NewUnlockErrorFull 

func NewUnlockErrorFull(t Transient, initial *states.State) Full

NewUnlockErrorFull returns a state manager that is useful for testing errors (mostly Unlock errors) when used with the clistate.Locker interface. Lock() does not return an error because clistate.Locker Lock()s the state at the start of Unlock(), so Lock() must succeeded for Unlock() to get called.

type LockDisabled 

type LockDisabled struct {
	// We can't embed State directly since Go dislikes that a field is
	// State and State interface has a method State
	Inner Full
}

LockDisabled implements State and Locker but disables state locking. If State doesn't support locking, this is a no-op. This is useful for easily disabling locking of an existing state or for tests.

func (*LockDisabled) GetRootOutputValues 

func (s *LockDisabled) GetRootOutputValues() (map[string]*states.OutputValue, error)

func (*LockDisabled) Lock 

func (s *LockDisabled) Lock(info *LockInfo) (string, error)

func (*LockDisabled) PersistState 

func (s *LockDisabled) PersistState(schemas *tofu.Schemas) error

func (*LockDisabled) RefreshState 

func (s *LockDisabled) RefreshState() error

func (*LockDisabled) State 

func (s *LockDisabled) State() *states.State

func (*LockDisabled) Unlock 

func (s *LockDisabled) Unlock(id string) error

func (*LockDisabled) WriteState 

func (s *LockDisabled) WriteState(v *states.State) error

type LockError 

type LockError struct {
	Info *LockInfo
	Err  error

	// Set when writing of lock file fails because of conflict and
	// then reading fails because file doesn't exist (removed by other process)
	InconsistentRead bool
}

LockError is a specialization of type error that is returned by Locker.Lock to indicate that the lock is already held by another process and that retrying may be productive to take the lock once the other process releases it.

func (*LockError) Error 

func (e *LockError) Error() string

func (*LockError) Retriable 

func (e *LockError) Retriable() bool

Retriable returns true when locking should be retried

func (*LockError) RetriableWithoutDelay 

func (e *LockError) RetriableWithoutDelay() bool

RetriableWithoutDelay returns true when delaying can be avoided

type LockInfo 

type LockInfo struct {
	// Unique ID for the lock. NewLockInfo provides a random ID, but this may
	// be overridden by the lock implementation. The final value of ID will be
	// returned by the call to Lock.
	ID string

	// OpenTofu operation, provided by the caller.
	Operation string

	// Extra information to store with the lock, provided by the caller.
	Info string

	// user@hostname when available
	Who string

	// OpenTofu version
	Version string

	// Time that the lock was taken.
	Created time.Time

	// Path to the state file when applicable. Set by the Lock implementation.
	Path string
}

LockInfo stores lock metadata.

Only Operation and Info are required to be set by the caller of Lock. Most callers should use NewLockInfo to create a LockInfo value with many of the fields populated with suitable default values.

func NewLockInfo 

func NewLockInfo() *LockInfo

NewLockInfo creates a LockInfo object and populates many of its fields with suitable default values.

func (*LockInfo) Err 

func (l *LockInfo) Err() error

Err returns the lock info formatted in an error

func (*LockInfo) Marshal 

func (l *LockInfo) Marshal() []byte

Marshal returns a string json representation of the LockInfo

func (*LockInfo) String 

func (l *LockInfo) String() string

String return a multi-line string representation of LockInfo

type Locker 

type Locker interface {
	// Lock attempts to obtain a lock, using the given lock information.
	//
	// The result is an opaque id that can be passed to Unlock to release
	// the lock, or an error if the lock cannot be acquired. Lock returns
	// an instance of LockError immediately if the lock is already held,
	// and the helper function LockWithContext uses this to automatically
	// retry lock acquisition periodically until a timeout is reached.
	Lock(info *LockInfo) (string, error)

	// Unlock releases a lock previously acquired by Lock.
	//
	// If the lock cannot be released -- for example, if it was stolen by
	// another user with some sort of administrative override privilege --
	// then an error is returned explaining the situation in a way that
	// is suitable for returning to an end-user.
	Unlock(id string) error
}

Locker is the interface for state managers that are able to manage mutual-exclusion locks for state.

Implementing Locker alongside Persistent relaxes some of the usual implementation constraints for implementations of Refresher and Persister, under the assumption that the locking mechanism effectively prevents multiple OpenTofu processes from reading and writing state concurrently. In particular, a type that implements both Locker and Persistent is only required to that the Persistent implementation is concurrency-safe within a single OpenTofu process.

A Locker implementation must ensure that another processes with a similarly-configured state manager cannot successfully obtain a lock while the current process is holding it, or vice-versa, assuming that both processes agree on the locking mechanism.

A Locker is not required to prevent non-cooperating processes from concurrently modifying the state, but is free to do so as an extra protection. If a mandatory locking mechanism of this sort is implemented, the state manager must ensure that RefreshState and PersistState calls can succeed if made through the same manager instance that is holding the lock, such has by retaining some sort of lock token that the Persistent methods can then use.

type Migrator 

type Migrator interface {
	PersistentMeta

	// StateForMigration returns a full statefile representing the latest
	// snapshot (as would be returned by Reader.State) and the associated
	// snapshot metadata (as would be returned by
	// PersistentMeta.StateSnapshotMeta).
	//
	// Just as with Reader.State, this must not fail.
	StateForMigration() *statefile.File

	// WriteStateForMigration accepts a full statefile including associated
	// snapshot metadata, and atomically updates the stored file (as with
	// Writer.WriteState) and the metadata.
	//
	// If "force" is not set, the manager must call CheckValidImport with
	// the given file and the current file and complete the update only if
	// that function returns nil. If force is set this may override such
	// checks, but some backends do not support forcing and so will act
	// as if force is always false.
	WriteStateForMigration(f *statefile.File, force bool) error
}

Migrator is an optional interface implemented by state managers that are capable of direct migration of state snapshots with their associated metadata unchanged.

This interface is used when available by function Migrate. See that function for more information on how it is used.

type OutputReader 

type OutputReader interface {
	// GetRootOutputValues fetches the root module output values from state or another source
	GetRootOutputValues() (map[string]*states.OutputValue, error)
}

OutputReader is the interface for managers that fetches output values from state or another source. This is a refinement of fetching the entire state and digging the output values from it because enhanced backends can apply special permissions to differentiate reading the state and reading the outputs within the state.

type Persistent 

type Persistent interface {
	Refresher
	Persister
	OutputReader
}

Persistent is a union of the Refresher and Persistent interfaces, for types that deal with persistent snapshots.

Persistent snapshots are ones that are retained in storage that will outlive a particular OpenTofu process, and are shared with other OpenTofu processes that have a similarly-configured state manager.

A manager may also choose to retain historical persistent snapshots, but that is an implementation detail and not visible via this API.

type PersistentMeta 

type PersistentMeta interface {
	// StateSnapshotMeta returns metadata about the state snapshot most
	// recently created either by a call to PersistState or read by a call
	// to RefreshState.
	//
	// If no persistent snapshot is yet available in the manager then
	// the return value is meaningless. This method is primarily available
	// for testing and logging purposes, and is of little use otherwise.
	StateSnapshotMeta() SnapshotMeta
}

PersistentMeta is an optional extension to Persistent that allows inspecting the metadata associated with the snapshot that was most recently either read by RefreshState or written by PersistState.

type Persister 

type Persister interface {
	PersistState(*tofu.Schemas) error
}

Persister is the interface for managers that can write snapshots to persistent storage.

Persister is usually implemented in conjunction with Writer, with PersistState copying the latest transient snapshot to be the new latest persistent snapshot.

A Persister implementation must detect updates made by other processes that may be running concurrently and avoid destroying those changes. This is most commonly achieved by making use of atomic write capabilities on the remote storage backend in conjunction with book-keeping with the Serial and Lineage fields in the standard state file formats.

Some implementations may optionally utilize config schema to persist state. For example, when representing state in an external JSON representation.

type Reader 

type Reader interface {
	// State returns the latest state.
	//
	// Each call to State returns an entirely-distinct copy of the state, with
	// no storage shared with any other call, so the caller may freely mutate
	// the returned object via the state APIs.
	State() *states.State
}

Reader is the interface for managers that can return transient snapshots of state.

Retrieving the snapshot must not fail, so retrieving a snapshot from remote storage (for example) should be dealt with elsewhere, often in an implementation of Refresher. For a type that implements both Reader and Refresher, it is okay for State to return nil if called before a RefreshState call has completed.

For a type that implements both Reader and Writer, State must return the result of the most recently completed call to WriteState, and the state manager must accept concurrent calls to both State and WriteState.

Each caller of this function must get a distinct copy of the state, and it must also be distinct from any instance cached inside the reader, to ensure that mutations of the returned state will not affect the values returned to other callers.

type Refresher 

type Refresher interface {
	// RefreshState retrieves a snapshot of state from persistent storage,
	// returning an error if this is not possible.
	//
	// Types that implement RefreshState generally also implement a State
	// method that returns the result of the latest successful refresh.
	//
	// Since only a subset of the data in a state is included when persisting,
	// a round-trip through PersistState and then RefreshState will often
	// return only a subset of what was written. Callers must assume that
	// ephemeral portions of the state may be unpopulated after calling
	// RefreshState.
	RefreshState() error
}

Refresher is the interface for managers that can read snapshots from persistent storage.

Refresher is usually implemented in conjunction with Reader, with RefreshState copying the latest persistent snapshot into the latest transient snapshot.

For a type that implements both Refresher and Persister, RefreshState must return the result of the most recently completed successful call to PersistState, unless another concurrently-running process has persisted another snapshot in the mean time.

The Refresher implementation must guarantee that the snapshot is read from persistent storage in a way that is safe under concurrent calls to PersistState that may be happening in other processes.

type SnapshotMeta 

type SnapshotMeta struct {
	// Lineage and Serial can be used to understand the relationships between
	// snapshots.
	//
	// If two snapshots both have an identical, non-empty Lineage
	// then the one with the higher Serial is newer than the other.
	// If the Lineage values are different or empty then the two snapshots
	// are unrelated and cannot be compared for relative age.
	Lineage string
	Serial  uint64

	// TerraformVersion is the number of the version of OpenTofu that created
	// the snapshot.
	TerraformVersion *version.Version
}

SnapshotMeta contains metadata about a persisted state snapshot.

This metadata is usually (but not necessarily) included as part of the "header" of a state file, which is then written to a raw blob storage medium by a persistent state manager.

Not all state managers will have useful values for all fields in this struct, so SnapshotMeta values are of little use beyond testing and logging use-cases.

func (SnapshotMeta) Compare 

func (m SnapshotMeta) Compare(existing SnapshotMeta) SnapshotMetaRel

Compare determines the relationship, if any, between the given existing SnapshotMeta and the potential "new" SnapshotMeta that is the receiver.

type SnapshotMetaRel 

type SnapshotMetaRel rune

SnapshotMetaRel describes a relationship between two SnapshotMeta values, returned from the SnapshotMeta.Compare method where the "first" value is the receiver of that method and the "second" is the given argument. 

const (
	// SnapshotOlder indicates that two snapshots have a common lineage and
	// that the first has a lower serial value.
	SnapshotOlder SnapshotMetaRel = '<'

	// SnapshotNewer indicates that two snapshots have a common lineage and
	// that the first has a higher serial value.
	SnapshotNewer SnapshotMetaRel = '>'

	// SnapshotEqual indicates that two snapshots have a common lineage and
	// the same serial value.
	SnapshotEqual SnapshotMetaRel = '='

	// SnapshotUnrelated indicates that two snapshots have different lineage
	// and thus cannot be meaningfully compared.
	SnapshotUnrelated SnapshotMetaRel = '!'

	// SnapshotLegacy indicates that one or both of the snapshots
	// does not have a lineage at all, and thus no comparison is possible.
	SnapshotLegacy SnapshotMetaRel = '?'
)

func (SnapshotMetaRel) String 

func (i SnapshotMetaRel) String() string

type Storage 

type Storage interface {
	Transient
	Persistent
}

Storage is the union of Transient and Persistent, for state managers that have both transient and persistent storage.

Types implementing this interface coordinate between their Transient and Persistent implementations so that the persistent operations read or write the transient store.

type Transient 

type Transient interface {
	Reader
	Writer
}

Transient is a union of the Reader and Writer interfaces, for types that deal with transient snapshots.

Transient snapshots are ones that are generally retained only locally and to not create any historical version record when updated. Transient snapshots are not expected to outlive a particular OpenTofu process, and are not shared with any other process.

A state manager type that is primarily concerned with persistent storage may embed type Transient and then call State from its PersistState and WriteState from its RefreshState in order to build on any existing Transient implementation, such as the one returned by NewTransientInMemory.

func NewTransientInMemory 

func NewTransientInMemory(initial *states.State) Transient

NewTransientInMemory returns a Transient implementation that retains transient snapshots only in memory, as part of the object.

The given initial state, if any, must not be modified concurrently while this function is running, but may be freely modified once this function returns without affecting the stored transient snapshot.

type Writer 

type Writer interface {
	// WriteState saves a transient snapshot of the given state.
	//
	// The caller must ensure that the given state object is not concurrently
	// modified while a WriteState call is in progress. WriteState itself
	// will never modify the given state.
	WriteState(*states.State) error
}

Writer is the interface for managers that can create transient snapshots from state.

Writer is the opposite of Reader, and so it must update whatever the State method reads from. It does not write the state to any persistent storage, and (for managers that support historical versions) must not be recorded as a persistent new version of state.

Implementations that cache the state in memory must take a deep copy of it, since the caller may continue to modify the given state object after WriteState returns.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.