etcd

package
v1.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 31, 2022 License: Apache-2.0 Imports: 36 Imported by: 0

Documentation

Overview

Package metadata stores all labels and object specific metadata by namespace. This package also contains the main garbage collection logic for cleaning up resources consistently and atomically. Resources used by backends will be tracked in the metadata store to be exposed to consumers of this package.

The layout where a "/" delineates a bucket is described in the following section. Please try to follow this as closely as possible when adding functionality. We can bolster this with helpers and more structure if that becomes an issue.

Generically, we try to do the following: 

<version>/<namespace>/<object>/<key> -> <field>

version: Currently, this is "v1". Additions can be made to v1 in a backwards compatible way. If the layout changes, a new version must be made, along with a migration.

namespace: the namespace to which this object belongs.

object: defines which object set is stored in the bucket. There are two special objects, "labels" and "indexes". The "labels" bucket stores the labels for the parent namespace. The "indexes" object is reserved for indexing objects, if we require in the future.

key: object-specific key identifying the storage bucket for the objects contents.

Below is the current database schema. This should be updated each time the structure is changed in addition to adding a migration and incrementing the database version. Notes:

  • `╘══*...*` refers to maps with arbitrary keys

  • `version` is a key to a numeric value identifying the minor revisions of schema version

  • a namespace in a schema bucket cannot be named "version"

    └──v1 - Schema version bucket ├──version : <varint> - Latest version, see migrations ╘══*namespace* ├──labels │  ╘══*key* : <string> - Label value ├──image │  ╘══*image name* │   ├──createdat : <binary time> - Created at │   ├──updatedat : <binary time> - Updated at │   ├──target │   │  ├──digest : <digest> - Descriptor digest │   │  ├──mediatype : <string> - Descriptor media type │   │  └──size : <varint> - Descriptor size │   └──labels │   ╘══*key* : <string> - Label value ├──containers │  ╘══*container id* │   ├──createdat : <binary time> - Created at │   ├──updatedat : <binary time> - Updated at │   ├──spec : <binary> - Proto marshaled spec │   ├──image : <string> - Image name │   ├──snapshotter : <string> - Snapshotter name │   ├──snapshotKey : <string> - Snapshot key │   ├──runtime │   │  ├──name : <string> - Runtime name │   │  ├──extensions │   │  │  ╘══*name* : <binary> - Proto marshaled extension │   │  └──options : <binary> - Proto marshaled options │   └──labels │   ╘══*key* : <string> - Label value ├──snapshots │  ╘══*snapshotter* │   ╘══*snapshot key* │    ├──name : <string> - Snapshot name in backend │   ├──createdat : <binary time> - Created at │   ├──updatedat : <binary time> - Updated at │    ├──parent : <string> - Parent snapshot name │   ├──children │   │  ╘══*snapshot key* : <nil> - Child snapshot reference │   └──labels │   ╘══*key* : <string> - Label value ├──content │  ├──blob │  │ ╘══*blob digest* │  │ ├──createdat : <binary time> - Created at │  │ ├──updatedat : <binary time> - Updated at │  │   ├──size : <varint> - Blob size │  │ └──labels │  │ ╘══*key* : <string> - Label value │  └──ingests │   ╘══*ingest reference* │    ├──ref : <string> - Ingest reference in backend │   ├──expireat : <binary time> - Time to expire ingest │   └──expected : <digest> - Expected commit digest └──leases ╘══*lease id* ├──createdat : <binary time> - Created at ├──labels │ ╘══*key* : <string> - Label value ├──snapshots │  ╘══*snapshotter* │   ╘══*snapshot key* : <nil> - Snapshot reference ├──content │  ╘══*blob digest* : <nil> - Content blob reference └──ingests ╘══*ingest reference* : <nil> - Content ingest reference

Index

Constants

View Source 
const (
	// ResourceUnknown specifies an unknown resource
	ResourceUnknown gc.ResourceType = iota
	// ResourceContent specifies a content resource
	ResourceContent
	// ResourceSnapshot specifies a snapshot resource
	ResourceSnapshot
	// ResourceContainer specifies a container resource
	ResourceContainer
	// ResourceTask specifies a task resource
	ResourceTask
	// ResourceLease specifies a lease
	ResourceLease
	// ResourceIngest specifies a content ingest
	ResourceIngest
)

Variables

View Source 
var (
	KeyVersion          = schemaVersion
	KeyDBVersion        = "version"    // stores the version of the schema
	KeyObjectLabels     = "labels"     // stores the labels for a namespace.
	KeyObjectImages     = "images"     // stores image objects
	KeyObjectContainers = "containers" // stores container objects
	KeyObjectSnapshots  = "snapshots"  // stores snapshot references
	KeyObjectContent    = "content"    // stores content references
	KeyObjectBlob       = "blob"       // stores content links
	KeyObjectIngests    = "ingests"    // stores ingest objects
	KeyObjectLeases     = "leases"     // stores leases
	KeyObjectSandboxes  = "sandboxes"  // stores sandboxes

	KeyDigest      = "digest"
	KeyMediaType   = "mediatype"
	KeySize        = "size"
	KeyImage       = "image"
	KeyRuntime     = "runtime"
	KeyName        = "name"
	KeyParent      = "parent"
	KeyChildren    = "children"
	KeyOptions     = "options"
	KeySpec        = "spec"
	KeySnapshotKey = "snapshotKey"
	KeySnapshotter = "snapshotter"
	KeyTarget      = "target"
	KeyExtensions  = "extensions"
	KeyCreatedAt   = "createdat"
	KeyUpdatedAt   = "updatedat"
	KeyExpected    = "expected"
	KeyRef         = "ref"
	KeyExpireAt    = "expireat"
	KeySandboxID   = "sandboxid"

	// additional key for etcd plugin
	KeySchedulerVersion = "scheduler.v1"
	KeyDirtyCS          = "dirtyCS"
	KeyDirtySS          = "dirtySS"
	KeyDirty            = "dirty"

	KeyObjectNamespace = "namespaces"
	KeyLock            = "lock"
	KeyAnnotations     = "annotations"
	KeyNil             = "1"
	KeyUpperdir        = "upperdir"
	KeyLowerdir        = "lowerdir"
)
View Source 
var EnableEtcdGC bool
View Source 
var LeaseBackend string

Functions

func NewBoltLeaseManager 

func NewBoltLeaseManager(db *DB) leases.Manager

NewboltLeaseManager creates a new lease manager for managing leases using the provided database transaction.

func NewContainerStore 

func NewContainerStore(db *DB) containers.Store

NewContainerStore returns a Store backed by an underlying bolt DB

func NewImageStore 

func NewImageStore(db *DB) images.Store

NewImageStore returns a store backed by a bolt DB

func NewLeaseManager 

func NewLeaseManager(db *DB) leases.Manager

NewLeaseManager creates a new lease manager for managing leases using the provided database transaction.

func NewNamespaceStore 

func NewNamespaceStore(db *DB) namespaces.Store

func NewSandboxStore 

func NewSandboxStore(db *DB) api.Store

NewSandboxStore creates a datababase client for sandboxes

func WithPolicyIsolated 

func WithPolicyIsolated(o *dbOptions)

WithPolicyIsolated isolates contents between namespaces

func WithTransactionContext 

func WithTransactionContext(ctx context.Context, tx *bolt.Tx) context.Context

Types

type CollectionContext 

type CollectionContext interface {
	// Sends all known resources
	All(func(gc.Node))

	// Active sends all active resources
	// Leased resources may be excluded since lease ownership should take
	// precedence over active status.
	Active(namespace string, fn func(gc.Node))

	// Leased sends all resources associated with the given lease
	Leased(namespace, lease string, fn func(gc.Node))

	// Remove marks the given resource as removed
	Remove(gc.Node)

	// Cancel is called to cleanup a context after a failed collection
	Cancel() error

	// Finish is called to cleanup a context after a successful collection
	Finish() error
}

CollectionContext manages a resource collection during a single run of the garbage collector. The context is responsible for managing access to resources as well as tracking removal. Implementations should defer any longer running operations to the Finish function and optimize other functions for running fast during garbage collection write locks.

type Collector 

type Collector interface {
	StartCollection(context.Context) (CollectionContext, error)

	ReferenceLabel() string
}

Collector is an interface to manage resource collection for any collectible resource registered for garbage collection.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func NewDB 

func NewDB(c clientv3.Config, bdb *bolt.DB, cs content.Store, ss map[string]snapshots.Snapshotter, opts ...DBOpt) (*DB, error)

func (*DB) Close 

func (m *DB) Close() error

func (*DB) ContentStore 

func (m *DB) ContentStore() content.Store

ContentStore returns a namespaced content store proxied to a content store.

func (*DB) EUpdate 

func (m *DB) EUpdate(tx func(transactor) error) error

func (*DB) GarbageCollect 

func (m *DB) GarbageCollect(ctx context.Context) (gc.Stats, error)

GarbageCollect removes resources (snapshots, contents, ...) that are no longer used.

func (*DB) Get 

func (m *DB) Get(key string, opts ...clientv3.OpOption) (map[string]string, error)

func (*DB) Init 

func (m *DB) Init() error

func (*DB) Lock 

func (m *DB) Lock(pfx string) (func() error, error)

func (*DB) Put 

func (m *DB) Put(key, value string, opts ...clientv3.OpOption) error

func (*DB) RLock 

func (m *DB) RLock(pfx string) (func() error, error)

func (*DB) RegisterCollectibleResource 

func (m *DB) RegisterCollectibleResource(t gc.ResourceType, c Collector)

RegisterCollectibleResource registers a resource type which can be referenced by metadata resources and garbage collected. Collectible Resources are useful ephemeral resources which need to to be tracked by go away after reboot or process restart.

A few limitations to consider:

  • Collectible Resources cannot reference other resources.
  • A failure to complete collection will not fail the garbage collection, however, the resources can be collected in a later run.
  • Collectible Resources must track whether the resource is active and/or lease membership.

func (*DB) RegisterMutationCallback 

func (m *DB) RegisterMutationCallback(fn func(bool))

RegisterMutationCallback registers a function to be called after a metadata mutations has been performed.

The callback function is an argument for whether a deletion has occurred since the last garbage collection.

func (*DB) Snapshotter 

func (m *DB) Snapshotter(name string) snapshots.Snapshotter

Snapshotter returns a snapshotter for the requested snapshotter name proxied to a snapshotter.

func (*DB) Snapshotters 

func (m *DB) Snapshotters() map[string]snapshots.Snapshotter

Snapshotters returns all available snapshotters.

func (*DB) TryLock 

func (m *DB) TryLock(pfx string) (func() error, error)

func (*DB) Update 

func (m *DB) Update(fn func(*bolt.Tx) error) error

Update runs a writable transaction on the metadata store.

func (*DB) View 

func (m *DB) View(fn func(*bolt.Tx) error) error

View runs a readonly transaction on the metadata store.

func (*DB) WithTransactionContext 

func (m *DB) WithTransactionContext(ctx context.Context, tx *bolt.Tx) context.Context

WithTransactionContext returns a new context holding the provided bolt transaction. Functions which require a bolt transaction will first check to see if a transaction is already created on the context before creating their own.

type DBOpt 

type DBOpt func(*dbOptions)

DBOpt configures how we set up the DB

type GCStats 

type GCStats struct {
	MetaD     time.Duration
	ContentD  time.Duration
	SnapshotD map[string]time.Duration
}

GCStats holds the duration for the different phases of the garbage collector

func (GCStats) Elapsed 

func (s GCStats) Elapsed() time.Duration

Elapsed returns the duration which elapsed during a collection

type Transactor 

type Transactor struct {
	// contains filtered or unexported fields
}

func NewTransactor 

func NewTransactor(db *DB) *Transactor

func (*Transactor) Get 

func (t *Transactor) Get(key string, opts ...clientv3.OpOption) (map[string]string, error)

func (*Transactor) Lock 

func (t *Transactor) Lock(pfx string) (func() error, error)

func (*Transactor) Put 

func (t *Transactor) Put(key, value string, opts ...clientv3.OpOption) error

func (*Transactor) RLock 

func (t *Transactor) RLock(pfx string) (func() error, error)

func (*Transactor) TryLock 

func (t *Transactor) TryLock(pfx string) (func() error, error)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.