metadata

Documentation

Overview

Package metadata stores all labels and object specific metadata by namespace. This package also contains the main garbage collection logic for cleaning up resources consistently and atomically. Resources used by backends will be tracked in the metadata store to be exposed to consumers of this package.

The layout where a "/" delineates a bucket is described in the following section. Please try to follow this as closely as possible when adding functionality. We can bolster this with helpers and more structure if that becomes an issue.

Generically, we try to do the following:

<version>/<namespace>/<object>/<key> -> <field>

version: Currently, this is "v1". Additions can be made to v1 in a backwards compatible way. If the layout changes, a new version must be made, along with a migration.

namespace: the namespace to which this object belongs.

object: defines which object set is stored in the bucket. There are two special objects, "labels" and "indexes". The "labels" bucket stores the labels for the parent namespace. The "indexes" object is reserved for indexing objects, if we require in the future.

key: object-specific key identifying the storage bucket for the objects contents.

Below is the current database schema. This should be updated each time the structure is changed in addition to adding a migration and incrementing the database version. Notes:

• `╘══*...*` refers to maps with arbitrary keys

• `version` is a key to a numeric value identifying the minor revisions of schema version

• a namespace in a schema bucket cannot be named "version"

  └──v1 - Schema version bucket ├──version : <varint> - Latest version, see migrations ╘══*namespace* ├──labels │  ╘══*key* : <string> - Label value ├──image │  ╘══*image name* │   ├──createdat : <binary time> - Created at │   ├──updatedat : <binary time> - Updated at │   ├──target │   │  ├──digest : <digest> - Descriptor digest │   │  ├──mediatype : <string> - Descriptor media type │   │  └──size : <varint> - Descriptor size │   └──labels │   ╘══*key* : <string> - Label value ├──containers │  ╘══*container id* │   ├──createdat : <binary time> - Created at │   ├──updatedat : <binary time> - Updated at │   ├──spec : <binary> - Proto marshaled spec │   ├──image : <string> - Image name │   ├──snapshotter : <string> - Snapshotter name │   ├──snapshotKey : <string> - Snapshot key │   ├──runtime │   │  ├──name : <string> - Runtime name │   │  ├──extensions │   │  │  ╘══*name* : <binary> - Proto marshaled extension │   │  └──options : <binary> - Proto marshaled options │   └──labels │   ╘══*key* : <string> - Label value ├──snapshots │  ╘══*snapshotter* │   ╘══*snapshot key* │    ├──name : <string> - Snapshot name in backend │   ├──createdat : <binary time> - Created at │   ├──updatedat : <binary time> - Updated at │    ├──parent : <string> - Parent snapshot name │   ├──children │   │  ╘══*snapshot key* : <nil> - Child snapshot reference │   └──labels │   ╘══*key* : <string> - Label value ├──content │  ├──blob │  │ ╘══*blob digest* │  │ ├──createdat : <binary time> - Created at │  │ ├──updatedat : <binary time> - Updated at │  │   ├──size : <varint> - Blob size │  │ └──labels │  │ ╘══*key* : <string> - Label value │  └──ingests │   ╘══*ingest reference* │    ├──ref : <string> - Ingest reference in backend │   ├──expireat : <binary time> - Time to expire ingest │   └──expected : <digest> - Expected commit digest └──leases ╘══*lease id* ├──createdat : <binary time> - Created at ├──labels │ ╘══*key* : <string> - Label value ├──snapshots │  ╘══*snapshotter* │   ╘══*snapshot key* : <nil> - Snapshot reference ├──content │  ╘══*blob digest* : <nil> - Content blob reference └──ingests ╘══*ingest reference* : <nil> - Content ingest reference

Index

• Constants
• func NewContainerStore(db *DB) containers.Store
• func NewImageStore(db *DB) images.Store
• func NewLeaseManager(db *DB) leases.Manager
• func NewNamespaceStore(tx *bolt.Tx) namespaces.Store
• func NewSandboxStore(db *DB) api.Store
• func WithPolicyIsolated(o *dbOptions)
• func WithTransactionContext(ctx context.Context, tx *bolt.Tx) context.Context
• type CollectionContext
• type Collector
• type DB
  • func NewDB(db *bolt.DB, cs content.Store, ss map[string]snapshots.Snapshotter, ...) *DB
  • func (m *DB) ContentStore() content.Store
  • func (m *DB) GarbageCollect(ctx context.Context) (gc.Stats, error)
  • func (m *DB) Init(ctx context.Context) error
  • func (m *DB) RegisterCollectibleResource(t gc.ResourceType, c Collector)
  • func (m *DB) RegisterMutationCallback(fn func(bool))
  • func (m *DB) Snapshotter(name string) snapshots.Snapshotter
  • func (m *DB) Snapshotters() map[string]snapshots.Snapshotter
  • func (m *DB) Update(fn func(*bolt.Tx) error) error
  • func (m *DB) View(fn func(*bolt.Tx) error) error
• type DBOpt
  • func WithEventsPublisher(p events.Publisher) DBOpt
• type GCStats
  • func (s GCStats) Elapsed() time.Duration

Constants

const (
	// ResourceUnknown specifies an unknown resource
	ResourceUnknown gc.ResourceType = iota
	// ResourceContent specifies a content resource
	ResourceContent
	// ResourceSnapshot specifies a snapshot resource
	ResourceSnapshot
	// ResourceContainer specifies a container resource
	ResourceContainer
	// ResourceTask specifies a task resource
	ResourceTask
	// ResourceLease specifies a lease
	ResourceLease
	// ResourceIngest specifies a content ingest
	ResourceIngest

	// ResourceStream specifies a stream
	ResourceStream
)

Functions

func NewContainerStore

func NewContainerStore(db *DB) containers.Store

NewContainerStore returns a Store backed by an underlying bolt DB

func NewImageStore

func NewImageStore(db *DB) images.Store

NewImageStore returns a store backed by a bolt DB

func NewLeaseManager

func NewLeaseManager(db *DB) leases.Manager

NewLeaseManager creates a new lease manager for managing leases using the provided database transaction.

func NewNamespaceStore

func NewNamespaceStore(tx *bolt.Tx) namespaces.Store

NewNamespaceStore returns a store backed by a bolt DB

func NewSandboxStore

func NewSandboxStore(db *DB) api.Store

NewSandboxStore creates a datababase client for sandboxes

func WithPolicyIsolated

func WithPolicyIsolated(o *dbOptions)

WithPolicyIsolated isolates contents between namespaces

func WithTransactionContext

func WithTransactionContext(ctx context.Context, tx *bolt.Tx) context.Context

WithTransactionContext returns a new context holding the provided bolt transaction. Functions which require a bolt transaction will first check to see if a transaction is already created on the context before creating their own.

Types

type CollectionContext

type CollectionContext interface {
	// Sends all known resources
	All(func(gc.Node))

	// Active sends all active resources
	// Leased resources may be excluded since lease ownership should take
	// precedence over active status.
	Active(namespace string, fn func(gc.Node))

	// Leased sends all resources associated with the given lease
	Leased(namespace, lease string, fn func(gc.Node))

	// Remove marks the given resource as removed
	Remove(gc.Node)

	// Cancel is called to cleanup a context after a failed collection
	Cancel() error

	// Finish is called to cleanup a context after a successful collection
	Finish() error
}

CollectionContext manages a resource collection during a single run of the garbage collector. The context is responsible for managing access to resources as well as tracking removal. Implementations should defer any longer running operations to the Finish function and optimize other functions for running fast during garbage collection write locks.

type Collector

type Collector interface {
	StartCollection(context.Context) (CollectionContext, error)

	ReferenceLabel() string
}

Collector is an interface to manage resource collection for any collectible resource registered for garbage collection.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB represents a metadata database backed by a bolt database. The database is fully namespaced and stores image, container, namespace, snapshot, and content data while proxying data shared across namespaces to backend datastores for content and snapshots.

func NewDB

func NewDB(db *bolt.DB, cs content.Store, ss map[string]snapshots.Snapshotter, opts ...DBOpt) *DB

NewDB creates a new metadata database using the provided bolt database, content store, and snapshotters.

func (*DB) ContentStore

func (m *DB) ContentStore() content.Store

ContentStore returns a namespaced content store proxied to a content store.

func (*DB) GarbageCollect

func (m *DB) GarbageCollect(ctx context.Context) (gc.Stats, error)

GarbageCollect removes resources (snapshots, contents, ...) that are no longer used.

func (*DB) Init

func (m *DB) Init(ctx context.Context) error

Init ensures the database is at the correct version and performs any needed migrations.

func (*DB) RegisterCollectibleResource

func (m *DB) RegisterCollectibleResource(t gc.ResourceType, c Collector)

RegisterCollectibleResource registers a resource type which can be referenced by metadata resources and garbage collected. Collectible Resources are useful ephemeral resources which need to be tracked by go away after reboot or process restart.

A few limitations to consider:

• Collectible Resources cannot reference other resources.
• A failure to complete collection will not fail the garbage collection, however, the resources can be collected in a later run.
• Collectible Resources must track whether the resource is active and/or lease membership.

func (*DB) RegisterMutationCallback

func (m *DB) RegisterMutationCallback(fn func(bool))

RegisterMutationCallback registers a function to be called after a metadata mutations has been performed.

The callback function is an argument for whether a deletion has occurred since the last garbage collection.

func (*DB) Snapshotter

func (m *DB) Snapshotter(name string) snapshots.Snapshotter

Snapshotter returns a snapshotter for the requested snapshotter name proxied to a snapshotter.

func (*DB) Snapshotters

func (m *DB) Snapshotters() map[string]snapshots.Snapshotter

Snapshotters returns all available snapshotters.

func (*DB) Update

func (m *DB) Update(fn func(*bolt.Tx) error) error

Update runs a writable transaction on the metadata store.

func (*DB) View

func (m *DB) View(fn func(*bolt.Tx) error) error

View runs a readonly transaction on the metadata store.

type DBOpt

type DBOpt func(*dbOptions)

DBOpt configures how we set up the DB

func WithEventsPublisher

func WithEventsPublisher(p events.Publisher) DBOpt

WithEventsPublisher adds an events publisher to the metadata db to directly publish events

type GCStats

type GCStats struct {
	MetaD     time.Duration
	ContentD  time.Duration
	SnapshotD map[string]time.Duration
}

GCStats holds the duration for the different phases of the garbage collector

func (GCStats) Elapsed

func (s GCStats) Elapsed() time.Duration

Elapsed returns the duration which elapsed during a collection