distribution

README

The toolset to pack, ship, store, and deliver content.

This repository's main product is the Open Source Registry implementation for storing and distributing container images using the OCI Distribution Specification. The goal of this project is to provide a simple, secure, and scalable base for building a large scale registry solution or running a simple private registry. It is a core library for many registry operators including Docker Hub, GitHub Container Registry, GitLab Container Registry and DigitalOcean Container Registry, as well as the CNCF Harbor Project, and VMware Harbor Registry.

This repository contains the following components:

Component	Description
registry	An implementation of the OCI Distribution Specification.
libraries	A rich set of libraries for interacting with distribution components. Please see godoc for details. Note: The interfaces for these libraries are unstable.
documentation	Docker's full documentation set is available at docs.docker.com. This repository contains the subset related just to the registry.

How does this integrate with Docker, containerd, and other OCI client?

Clients implement against the OCI specification and communicate with the registry using HTTP. This project contains a client implementation which is currently in use by Docker, however, it is deprecated for the implementation in containerd and will not support new features.

What are the long term goals of the Distribution project?

The Distribution project has the further long term goal of providing a secure tool chain for distributing content. The specifications, APIs and tools should be as useful with Docker as they are without.

Our goal is to design a professional grade and extensible content distribution system that allow users to:

• Enjoy an efficient, secured and reliable way to store, manage, package and exchange content
• Hack/roll their own on top of healthy open-source components
• Implement their own home made solution through good specs, and solid extensions mechanism.

Contribution

Please see CONTRIBUTING.md for details on how to contribute issues, fixes, and patches to this project. If you are contributing code, see the instructions for building a development environment.

Communication

For async communication and long running discussions please use issues and pull requests on the github repo. This will be the best place to discuss design and implementation.

For sync communication we have a #distribution channel in the CNCF Slack that everyone is welcome to join and chat about development.

Licenses

The distribution codebase is released under the Apache 2.0 license. The README.md file, and files in the "docs" folder are licensed under the Creative Commons Attribution 4.0 International License. You may obtain a copy of the license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.

Documentation

Overview

Package distribution will define the interfaces for the components of docker distribution. The goal is to allow users to reliably package, ship and store content related to docker images.

This is currently a work in progress. More details are available in the README.md.

Index

• Variables
• func ManifestMediaTypes() (mediaTypes []string)
• func RegisterManifestSchema(mediaType string, u UnmarshalFunc) error
• func UnmarshalManifest(ctHeader string, p []byte) (Manifest, Descriptor, error)
• type BlobCreateOption
• type BlobDeleter
• type BlobDescriptorService
• type BlobDescriptorServiceFactory
• type BlobEnumerator
• type BlobIngester
• type BlobProvider
• type BlobServer
• type BlobService
• type BlobStatter
• type BlobStore
• type BlobWriter
• type CreateOptions
• type Describable
• type Descriptor
  • func (d Descriptor) Descriptor() Descriptor
• type ErrBlobInvalidDigest
  • func (err ErrBlobInvalidDigest) Error() string
• type ErrBlobMounted
  • func (err ErrBlobMounted) Error() string
• type ErrManifestBlobUnknown
  • func (err ErrManifestBlobUnknown) Error() string
• type ErrManifestNameInvalid
  • func (err ErrManifestNameInvalid) Error() string
• type ErrManifestUnknown
  • func (err ErrManifestUnknown) Error() string
• type ErrManifestUnknownRevision
  • func (err ErrManifestUnknownRevision) Error() string
• type ErrManifestUnverified
  • func (ErrManifestUnverified) Error() string
• type ErrManifestVerification
  • func (errs ErrManifestVerification) Error() string
• type ErrRepositoryNameInvalid
  • func (err ErrRepositoryNameInvalid) Error() string
• type ErrRepositoryUnknown
  • func (err ErrRepositoryUnknown) Error() string
• type ErrTagUnknown
  • func (err ErrTagUnknown) Error() string
• type Manifest
• type ManifestBuilder
• type ManifestEnumerator
• type ManifestService
• type ManifestServiceOption
  • func WithManifestMediaTypes(mediaTypes []string) ManifestServiceOption
  • func WithTag(tag string) ManifestServiceOption
• type Namespace
• type ReadSeekCloser
• type Repository
• type RepositoryEnumerator
• type RepositoryRemover
• type Scope
• type TagManifestsProvider
• type TagService
• type UnmarshalFunc
• type WithManifestMediaTypesOption
  • func (o WithManifestMediaTypesOption) Apply(m ManifestService) error
• type WithTagOption
  • func (o WithTagOption) Apply(m ManifestService) error

Variables

var (
	// ErrBlobExists returned when blob already exists
	ErrBlobExists = errors.New("blob exists")

	// ErrBlobDigestUnsupported when blob digest is an unsupported version.
	ErrBlobDigestUnsupported = errors.New("unsupported blob digest")

	// ErrBlobUnknown when blob is not found.
	ErrBlobUnknown = errors.New("unknown blob")

	// ErrBlobUploadUnknown returned when upload is not found.
	ErrBlobUploadUnknown = errors.New("blob upload unknown")

	// ErrBlobInvalidLength returned when the blob has an expected length on
	// commit, meaning mismatched with the descriptor or an invalid value.
	ErrBlobInvalidLength = errors.New("blob invalid length")
)

var ErrAccessDenied = errors.New("access denied")

ErrAccessDenied is returned when an access to a requested resource is denied.

var ErrManifestNotModified = errors.New("manifest not modified")

ErrManifestNotModified is returned when a conditional manifest GetByTag returns nil due to the client indicating it has the latest version

var ErrSchemaV1Unsupported = errors.New("manifest schema v1 unsupported")

ErrSchemaV1Unsupported is returned when a client tries to upload a schema v1 manifest but the registry is configured to reject it

var ErrUnsupported = errors.New("operation unsupported")

ErrUnsupported is returned when an unimplemented or unsupported action is performed

var GlobalScope = Scope(fullScope{})

GlobalScope represents the full namespace scope which contains all other scopes.

Functions

func ManifestMediaTypes

func ManifestMediaTypes() (mediaTypes []string)

ManifestMediaTypes returns the supported media types for manifests.

func RegisterManifestSchema

func RegisterManifestSchema(mediaType string, u UnmarshalFunc) error

RegisterManifestSchema registers an UnmarshalFunc for a given schema type. This should be called from specific

func UnmarshalManifest

func UnmarshalManifest(ctHeader string, p []byte) (Manifest, Descriptor, error)

UnmarshalManifest looks up manifest unmarshal functions based on MediaType

Types

type BlobCreateOption

type BlobCreateOption interface {
	Apply(interface{}) error
}

BlobCreateOption is a general extensible function argument for blob creation methods. A BlobIngester may choose to honor any or none of the given BlobCreateOptions, which can be specific to the implementation of the BlobIngester receiving them. TODO (brianbland): unify this with ManifestServiceOption in the future

type BlobDeleter

type BlobDeleter interface {
	Delete(ctx context.Context, dgst digest.Digest) error
}

BlobDeleter enables deleting blobs from storage.

type BlobDescriptorService

type BlobDescriptorService interface {
	BlobStatter

	// SetDescriptor assigns the descriptor to the digest. The provided digest and
	// the digest in the descriptor must map to identical content but they may
	// differ on their algorithm. The descriptor must have the canonical
	// digest of the content and the digest algorithm must match the
	// annotators canonical algorithm.
	//
	// Such a facility can be used to map blobs between digest domains, with
	// the restriction that the algorithm of the descriptor must match the
	// canonical algorithm (ie sha256) of the annotator.
	SetDescriptor(ctx context.Context, dgst digest.Digest, desc Descriptor) error

	// Clear enables descriptors to be unlinked
	Clear(ctx context.Context, dgst digest.Digest) error
}

BlobDescriptorService manages metadata about a blob by digest. Most implementations will not expose such an interface explicitly. Such mappings should be maintained by interacting with the BlobIngester. Hence, this is left off of BlobService and BlobStore.

type BlobDescriptorServiceFactory

type BlobDescriptorServiceFactory interface {
	BlobAccessController(svc BlobDescriptorService) BlobDescriptorService
}

BlobDescriptorServiceFactory creates middleware for BlobDescriptorService.

type BlobEnumerator

type BlobEnumerator interface {
	Enumerate(ctx context.Context, ingester func(dgst digest.Digest) error) error
}

BlobEnumerator enables iterating over blobs from storage

type BlobIngester

type BlobIngester interface {
	// Put inserts the content p into the blob service, returning a descriptor
	// or an error.
	Put(ctx context.Context, mediaType string, p []byte) (Descriptor, error)

	// Create allocates a new blob writer to add a blob to this service. The
	// returned handle can be written to and later resumed using an opaque
	// identifier. With this approach, one can Close and Resume a BlobWriter
	// multiple times until the BlobWriter is committed or cancelled.
	Create(ctx context.Context, options ...BlobCreateOption) (BlobWriter, error)

	// Resume attempts to resume a write to a blob, identified by an id.
	Resume(ctx context.Context, id string) (BlobWriter, error)
}

BlobIngester ingests blob data.

type BlobProvider

type BlobProvider interface {
	// Get returns the entire blob identified by digest along with the descriptor.
	Get(ctx context.Context, dgst digest.Digest) ([]byte, error)

	// Open provides an [io.ReadSeekCloser] to the blob identified by the provided
	// descriptor. If the blob is not known to the service, an error is returned.
	Open(ctx context.Context, dgst digest.Digest) (io.ReadSeekCloser, error)
}

BlobProvider describes operations for getting blob data.

type BlobServer

type BlobServer interface {
	// ServeBlob attempts to serve the blob, identified by dgst, via http. The
	// service may decide to redirect the client elsewhere or serve the data
	// directly.
	//
	// This handler only issues successful responses, such as 2xx or 3xx,
	// meaning it serves data or issues a redirect. If the blob is not
	// available, an error will be returned and the caller may still issue a
	// response.
	//
	// The implementation may serve the same blob from a different digest
	// domain. The appropriate headers will be set for the blob, unless they
	// have already been set by the caller.
	ServeBlob(ctx context.Context, w http.ResponseWriter, r *http.Request, dgst digest.Digest) error
}

BlobServer can serve blobs via http.

type BlobService

type BlobService interface {
	BlobStatter
	BlobProvider
	BlobIngester
}

BlobService combines the operations to access, read and write blobs. This can be used to describe remote blob services.

type BlobStatter

type BlobStatter interface {
	// Stat provides metadata about a blob identified by the digest. If the
	// blob is unknown to the describer, ErrBlobUnknown will be returned.
	Stat(ctx context.Context, dgst digest.Digest) (Descriptor, error)
}

BlobStatter makes blob descriptors available by digest. The service may provide a descriptor of a different digest if the provided digest is not canonical.

type BlobStore

type BlobStore interface {
	BlobService
	BlobServer
	BlobDeleter
}

BlobStore represent the entire suite of blob related operations. Such an implementation can access, read, write, delete and serve blobs.

type BlobWriter

type BlobWriter interface {
	io.WriteCloser
	io.ReaderFrom

	// Size returns the number of bytes written to this blob.
	Size() int64

	// ID returns the identifier for this writer. The ID can be used with the
	// Blob service to later resume the write.
	ID() string

	// StartedAt returns the time this blob write was started.
	StartedAt() time.Time

	// Commit completes the blob writer process. The content is verified
	// against the provided provisional descriptor, which may result in an
	// error. Depending on the implementation, written data may be validated
	// against the provisional descriptor fields. If MediaType is not present,
	// the implementation may reject the commit or assign "application/octet-
	// stream" to the blob. The returned descriptor may have a different
	// digest depending on the blob store, referred to as the canonical
	// descriptor.
	Commit(ctx context.Context, provisional Descriptor) (canonical Descriptor, err error)

	// Cancel ends the blob write without storing any data and frees any
	// associated resources. Any data written thus far will be lost. Cancel
	// implementations should allow multiple calls even after a commit that
	// result in a no-op. This allows use of Cancel in a defer statement,
	// increasing the assurance that it is correctly called.
	Cancel(ctx context.Context) error
}

BlobWriter provides a handle for inserting data into a blob store. Instances should be obtained from BlobWriteService.Writer and BlobWriteService.Resume. If supported by the store, a writer can be recovered with the id.

type CreateOptions

type CreateOptions struct {
	Mount struct {
		ShouldMount bool
		From        reference.Canonical
		// Stat allows to pass precalculated descriptor to link and return.
		// Blob access check will be skipped if set.
		Stat *Descriptor
	}
}

CreateOptions is a collection of blob creation modifiers relevant to general blob storage intended to be configured by the BlobCreateOption.Apply method.

type Describable

type Describable interface {
	Descriptor() Descriptor
}

Describable is an interface for descriptors

type Descriptor

type Descriptor struct {
	// MediaType describe the type of the content. All text based formats are
	// encoded as utf-8.
	MediaType string `json:"mediaType,omitempty"`

	// Digest uniquely identifies the content. A byte stream can be verified
	// against this digest.
	Digest digest.Digest `json:"digest,omitempty"`

	// Size in bytes of content.
	Size int64 `json:"size,omitempty"`

	// URLs contains the source URLs of this content.
	URLs []string `json:"urls,omitempty"`

	// Annotations contains arbitrary metadata relating to the targeted content.
	Annotations map[string]string `json:"annotations,omitempty"`

	// Platform describes the platform which the image in the manifest runs on.
	// This should only be used when referring to a manifest.
	Platform *v1.Platform `json:"platform,omitempty"`
}

Descriptor describes targeted content. Used in conjunction with a blob store, a descriptor can be used to fetch, store and target any kind of blob. The struct also describes the wire protocol format. Fields should only be added but never changed.

func (Descriptor) Descriptor

func (d Descriptor) Descriptor() Descriptor

Descriptor returns the descriptor, to make it satisfy the Describable interface. Note that implementations of Describable are generally objects which can be described, not simply descriptors; this exception is in place to make it more convenient to pass actual descriptors to functions that expect Describable objects.

type ErrBlobInvalidDigest

type ErrBlobInvalidDigest struct {
	Digest digest.Digest
	Reason error
}

ErrBlobInvalidDigest returned when digest check fails.

func (ErrBlobInvalidDigest) Error

func (err ErrBlobInvalidDigest) Error() string

type ErrBlobMounted

type ErrBlobMounted struct {
	From       reference.Canonical
	Descriptor Descriptor
}

ErrBlobMounted returned when a blob is mounted from another repository instead of initiating an upload session.

func (ErrBlobMounted) Error

func (err ErrBlobMounted) Error() string

type ErrManifestBlobUnknown

type ErrManifestBlobUnknown struct {
	Digest digest.Digest
}

ErrManifestBlobUnknown returned when a referenced blob cannot be found.

func (ErrManifestBlobUnknown) Error

func (err ErrManifestBlobUnknown) Error() string

type ErrManifestNameInvalid

type ErrManifestNameInvalid struct {
	Name   string
	Reason error
}

ErrManifestNameInvalid should be used to denote an invalid manifest name. Reason may set, indicating the cause of invalidity.

func (ErrManifestNameInvalid) Error

func (err ErrManifestNameInvalid) Error() string

type ErrManifestUnknown

type ErrManifestUnknown struct {
	Name string
	Tag  string
}

ErrManifestUnknown is returned if the manifest is not known by the registry.

func (ErrManifestUnknown) Error

func (err ErrManifestUnknown) Error() string

type ErrManifestUnknownRevision

type ErrManifestUnknownRevision struct {
	Name     string
	Revision digest.Digest
}

ErrManifestUnknownRevision is returned when a manifest cannot be found by revision within a repository.

func (ErrManifestUnknownRevision) Error

func (err ErrManifestUnknownRevision) Error() string

type ErrManifestUnverified

type ErrManifestUnverified struct{}

ErrManifestUnverified is returned when the registry is unable to verify the manifest.

func (ErrManifestUnverified) Error

func (ErrManifestUnverified) Error() string

type ErrManifestVerification

type ErrManifestVerification []error

ErrManifestVerification provides a type to collect errors encountered during manifest verification. Currently, it accepts errors of all types, but it may be narrowed to those involving manifest verification.

func (ErrManifestVerification) Error

func (errs ErrManifestVerification) Error() string

type ErrRepositoryNameInvalid

type ErrRepositoryNameInvalid struct {
	Name   string
	Reason error
}

ErrRepositoryNameInvalid should be used to denote an invalid repository name. Reason may set, indicating the cause of invalidity.

func (ErrRepositoryNameInvalid) Error

func (err ErrRepositoryNameInvalid) Error() string

type ErrRepositoryUnknown

type ErrRepositoryUnknown struct {
	Name string
}

ErrRepositoryUnknown is returned if the named repository is not known by the registry.

func (ErrRepositoryUnknown) Error

func (err ErrRepositoryUnknown) Error() string

type ErrTagUnknown

type ErrTagUnknown struct {
	Tag string
}

ErrTagUnknown is returned if the given tag is not known by the tag service

func (ErrTagUnknown) Error

func (err ErrTagUnknown) Error() string

type Manifest

type Manifest interface {
	// References returns a list of objects which make up this manifest.
	// A reference is anything which can be represented by a
	// distribution.Descriptor. These can consist of layers, resources or other
	// manifests.
	//
	// While no particular order is required, implementations should return
	// them from highest to lowest priority. For example, one might want to
	// return the base layer before the top layer.
	References() []Descriptor

	// Payload provides the serialized format of the manifest, in addition to
	// the media type.
	Payload() (mediaType string, payload []byte, err error)
}

Manifest represents a registry object specifying a set of references and an optional target

type ManifestBuilder

type ManifestBuilder interface {
	// Build creates the manifest from his builder.
	Build(ctx context.Context) (Manifest, error)

	// References returns a list of objects which have been added to this
	// builder. The dependencies are returned in the order they were added,
	// which should be from base to head.
	References() []Descriptor

	// AppendReference includes the given object in the manifest after any
	// existing dependencies. If the add fails, such as when adding an
	// unsupported dependency, an error may be returned.
	//
	// The destination of the reference is dependent on the manifest type and
	// the dependency type.
	AppendReference(dependency Describable) error
}

ManifestBuilder creates a manifest allowing one to include dependencies. Instances can be obtained from a version-specific manifest package. Manifest specific data is passed into the function which creates the builder.

type ManifestEnumerator

type ManifestEnumerator interface {
	// Enumerate calls ingester for each manifest.
	Enumerate(ctx context.Context, ingester func(digest.Digest) error) error
}

ManifestEnumerator enables iterating over manifests

type ManifestService

type ManifestService interface {
	// Exists returns true if the manifest exists.
	Exists(ctx context.Context, dgst digest.Digest) (bool, error)

	// Get retrieves the manifest specified by the given digest
	Get(ctx context.Context, dgst digest.Digest, options ...ManifestServiceOption) (Manifest, error)

	// Put creates or updates the given manifest returning the manifest digest
	Put(ctx context.Context, manifest Manifest, options ...ManifestServiceOption) (digest.Digest, error)

	// Delete removes the manifest specified by the given digest. Deleting
	// a manifest that doesn't exist will return ErrManifestNotFound
	Delete(ctx context.Context, dgst digest.Digest) error
}

ManifestService describes operations on image manifests.

type ManifestServiceOption

type ManifestServiceOption interface {
	Apply(ManifestService) error
}

ManifestServiceOption is a function argument for Manifest Service methods

func WithManifestMediaTypes

func WithManifestMediaTypes(mediaTypes []string) ManifestServiceOption

WithManifestMediaTypes lists the media types the client wishes the server to provide.

func WithTag

func WithTag(tag string) ManifestServiceOption

WithTag allows a tag to be passed into Put

type Namespace

type Namespace interface {
	// Scope describes the names that can be used with this Namespace. The
	// global namespace will have a scope that matches all names. The scope
	// effectively provides an identity for the namespace.
	Scope() Scope

	// Repository should return a reference to the named repository. The
	// registry may or may not have the repository but should always return a
	// reference.
	Repository(ctx context.Context, name reference.Named) (Repository, error)

	// Repositories fills 'repos' with a lexicographically sorted catalog of repositories
	// up to the size of 'repos' and returns the value 'n' for the number of entries
	// which were filled.  'last' contains an offset in the catalog, and 'err' will be
	// set to io.EOF if there are no more entries to obtain.
	Repositories(ctx context.Context, repos []string, last string) (n int, err error)

	// Blobs returns a blob enumerator to access all blobs
	Blobs() BlobEnumerator

	// BlobStatter returns a BlobStatter to control
	BlobStatter() BlobStatter
}

Namespace represents a collection of repositories, addressable by name. Generally, a namespace is backed by a set of one or more services, providing facilities such as registry access, trust, and indexing.

type ReadSeekCloser

type ReadSeekCloser = io.ReadSeekCloser

ReadSeekCloser is the primary reader type for blob data, combining io.ReadSeeker with io.Closer.

Deprecated: use io.ReadSeekCloser.

type Repository

type Repository interface {
	// Named returns the name of the repository.
	Named() reference.Named

	// Manifests returns a reference to this repository's manifest service.
	// with the supplied options applied.
	Manifests(ctx context.Context, options ...ManifestServiceOption) (ManifestService, error)

	// Blobs returns a reference to this repository's blob service.
	Blobs(ctx context.Context) BlobStore

	// Tags returns a reference to this repositories tag service
	Tags(ctx context.Context) TagService
}

Repository is a named collection of manifests and layers.

type RepositoryEnumerator

type RepositoryEnumerator interface {
	Enumerate(ctx context.Context, ingester func(string) error) error
}

RepositoryEnumerator describes an operation to enumerate repositories

type RepositoryRemover

type RepositoryRemover interface {
	Remove(ctx context.Context, name reference.Named) error
}

RepositoryRemover removes given repository

type Scope

type Scope interface {
	// Contains returns true if the name belongs to the namespace.
	Contains(name string) bool
}

Scope defines the set of items that match a namespace.

type TagManifestsProvider

type TagManifestsProvider interface {
	// ManifestDigests returns set of digests that this tag historically pointed to. This also
	// includes currently linked digest. There is no ordering guaranteed
	ManifestDigests(ctx context.Context, tag string) ([]digest.Digest, error)
}

TagManifestsProvider provides method to retrieve the digests of manifests that a tag historically pointed to

type TagService

type TagService interface {
	// Get retrieves the descriptor identified by the tag. Some
	// implementations may differentiate between "trusted" tags and
	// "untrusted" tags. If a tag is "untrusted", the mapping will be returned
	// as an ErrTagUntrusted error, with the target descriptor.
	Get(ctx context.Context, tag string) (Descriptor, error)

	// Tag associates the tag with the provided descriptor, updating the
	// current association, if needed.
	Tag(ctx context.Context, tag string, desc Descriptor) error

	// Untag removes the given tag association
	Untag(ctx context.Context, tag string) error

	// All returns the set of tags managed by this tag service
	All(ctx context.Context) ([]string, error)

	// Lookup returns the set of tags referencing the given digest.
	Lookup(ctx context.Context, digest Descriptor) ([]string, error)
}

TagService provides access to information about tagged objects.

type UnmarshalFunc

type UnmarshalFunc func([]byte) (Manifest, Descriptor, error)

UnmarshalFunc implements manifest unmarshalling a given MediaType

type WithManifestMediaTypesOption

type WithManifestMediaTypesOption struct{ MediaTypes []string }

WithManifestMediaTypesOption holds a list of accepted media types

func (WithManifestMediaTypesOption) Apply

func (o WithManifestMediaTypesOption) Apply(m ManifestService) error

Apply conforms to the ManifestServiceOption interface

type WithTagOption

type WithTagOption struct{ Tag string }

WithTagOption holds a tag

func (WithTagOption) Apply

func (o WithTagOption) Apply(m ManifestService) error

Apply conforms to the ManifestServiceOption interface