distribution

README

Notice: This repository hosts experimental components that are currently under heavy and fast-paced development, not-ready for public consumption. If you are looking for the stable registry, please head over to docker/docker-registry instead.

Distribution

The Docker toolset to pack, ship, store, and deliver content.

The main product of this repository is the new registry implementation for storing and distributing docker images. It supersedes the docker/docker- registry project with a new API design, focused around security and performance.

The Distribution project has the further long term goal of providing a secure tool chain for distributing content. The specifications, APIs and tools should be as useful with docker as they are without.

This repository contains the following components:

• registry (beta): An implementation of the Docker Registry HTTP API V2 for use with docker 1.5+.
• libraries (unstable): A rich set of libraries for interacting with distribution components. Please see godoc for details. Note that the libraries are not considered stable.
• dist (experimental): An experimental tool to provide distribution oriented functionality without the docker daemon.
• specifications: Distribution related specifications are available in doc/spec.
• documentation: Documentation is available in doc.

How will this integrate with Docker engine?

This project should provide an implementation to a V2 API for use in the Docker core project. The API should be embeddable and simplify the process of securely pulling and pushing content from docker daemons.

What are the long term goals of the Distribution project?

Design a professional grade and extensible content distribution system, that allow users to:

• Enjoy an efficient, secured and reliable way to store, manage, package and exchange content
• Hack/roll their own on top of healthy open-source components
• Implement their own home made solution through good specs, and solid extensions mechanism.

Features

The new registry implementation provides the following benefits:

• faster push and pull
• new, more efficient implementation
• simplified deployment
• pluggable storage backend
• webhook notifications

For information on upcoming functionality, please see ROADMAP.md.

Installation

TODO(stevvooe): Add the following here:

• docker file
• binary builds for non-docker environment (test installations, etc.)

Configuration

The registry server can be configured with a yaml file. The following is a simple example that can used for local development:

version: 0.1
loglevel: debug
storage:
    filesystem:
        rootdirectory: /tmp/registry-dev
http:
    addr: localhost:5000
    secret: asecretforlocaldevelopment
    debug:
        addr: localhost:5001

The above configures the registry instance to run on port 5000, binding to "localhost", with the debug server enabled. Registry data will be stored in "/tmp/registry-dev". Logging will be in "debug" mode, which is the most verbose.

A similar simple configuration is available at [cmd/registry/config.yml], which is generally useful for local development.

**TODO(stevvooe): Need a "best practice" configuration overview. Perhaps, we can point to a documentation section.

For full details about configuring a registry server, please see the documentation.

Upgrading

TODO: Add a section about upgrading from V1 registry along with link to migrating in documentation.

Build

If a go development environment is setup, one can use go get to install the registry command from the current latest:

go get github.com/docker/distribution/cmd/registry

The above will install the source repository into the GOPATH. The registry binary can then be run with the following:

$ $GOPATH/bin/registry -version
$GOPATH/bin/registry github.com/docker/distribution v2.0.0-alpha.1+unknown

The registry can be run with the default config using the following incantantation:

$ $GOPATH/bin/registry $GOPATH/src/github.com/docker/distribution/cmd/registry/config.yml
INFO[0000] endpoint local-8082 disabled, skipping        app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] endpoint local-8083 disabled, skipping        app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] listening on :5000                            app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] debug server listening localhost:5001

If it is working, one should see the above log messages.

Repeatable Builds

For the full development experience, one should cd into $GOPATH/src/github.com/docker/distribution. From there, the regular go commands, such as go test, should work per package (please see Developing if they don't work).

A Makefile has been provided as a convenience to support repeatable builds. Please install the following into GOPATH for it to work:

go get github.com/tools/godep github.com/golang/lint/golint

TODO(stevvooe): Add a make setup command to Makefile to run this. Have to think about how to interact with Godeps properly.

Once these commands are available in the GOPATH, run make to get a full build:

$ GOPATH=`godep path`:$GOPATH make
+ clean
+ fmt
+ vet
+ lint
+ build
github.com/docker/docker/vendor/src/code.google.com/p/go/src/pkg/archive/tar
github.com/Sirupsen/logrus
github.com/docker/libtrust
...
github.com/yvasiyarov/gorelic
github.com/docker/distribution/registry/handlers
github.com/docker/distribution/cmd/registry
+ test
...
ok    github.com/docker/distribution/digest 7.875s
ok    github.com/docker/distribution/manifest 0.028s
ok    github.com/docker/distribution/notifications  17.322s
?     github.com/docker/distribution/registry [no test files]
ok    github.com/docker/distribution/registry/api/v2  0.101s
?     github.com/docker/distribution/registry/auth  [no test files]
ok    github.com/docker/distribution/registry/auth/silly  0.011s
...
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry-api-descriptor-template
+ /Users/sday/go/src/github.com/docker/distribution/bin/dist
+ binaries

The above provides a repeatable build using the contents of the vendored Godeps directory. This includes formatting, vetting, linting, building, testing and generating tagged binaries. We can verify this worked by running the registry binary generated in the "./bin" directory:

$ ./bin/registry -version
./bin/registry github.com/docker/distribution v2.0.0-alpha.2-80-g16d8b2c.m

Developing

The above approaches are helpful for small experimentation. If more complex tasks are at hand, it is recommended to employ the full power of godep.

The Makefile is designed to have its GOPATH defined externally. This allows one to experiment with various development environment setups. This is primarily useful when testing upstream bugfixes, by modifying local code. This can be demonstrated using godep to migrate the GOPATH to use the specified dependencies. The GOPATH can be migrated to the current package versions declared in Godeps with the following command:

godep restore

WARNING: This command will checkout versions of the code specified in Godeps/Godeps.json, modifying the contents of GOPATH. If this is undesired, it is recommended to create a workspace devoted to work on the Distribution project.

With a successful run of the above command, one can now use make without specifying the GOPATH:

$ make

If that is successful, standard go commands, such as go test should work, per package, without issue.

Support

If any issues are encountered while using the Distribution project, several avenues are available for support:

IRC: #docker-distribution on FreeNode Issue Tracker: github.com/docker/distribution/issues Google Groups: https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution Mailing List: docker@dockerproject.org

Contribute

Please see CONTRIBUTING.md.

License

This project is distributed under Apache License, Version 2.0.

Documentation

Overview

Package distribution will define the interfaces for the components of docker distribution. The goal is to allow users to reliably package, ship and store content related to docker images.

This is currently a work in progress. More details are availalbe in the README.md.

Index

• Variables
• type Descriptor
• type ErrLayerInvalidDigest
  • func (err ErrLayerInvalidDigest) Error() string
• type ErrManifestUnknown
  • func (err ErrManifestUnknown) Error() string
• type ErrManifestUnverified
  • func (ErrManifestUnverified) Error() string
• type ErrManifestVerification
  • func (errs ErrManifestVerification) Error() string
• type ErrRepositoryNameInvalid
  • func (err ErrRepositoryNameInvalid) Error() string
• type ErrRepositoryUnknown
  • func (err ErrRepositoryUnknown) Error() string
• type ErrUnknownLayer
  • func (err ErrUnknownLayer) Error() string
• type ErrUnknownManifestRevision
  • func (err ErrUnknownManifestRevision) Error() string
• type Layer
• type LayerService
• type LayerUpload
• type ManifestService
• type Registry
• type Repository
• type SignatureService

Variables

var (
	// ErrLayerExists returned when layer already exists
	ErrLayerExists = fmt.Errorf("layer exists")

	// ErrLayerTarSumVersionUnsupported when tarsum is unsupported version.
	ErrLayerTarSumVersionUnsupported = fmt.Errorf("unsupported tarsum version")

	// ErrLayerUploadUnknown returned when upload is not found.
	ErrLayerUploadUnknown = fmt.Errorf("layer upload unknown")

	// ErrLayerClosed returned when an operation is attempted on a closed
	// Layer or LayerUpload.
	ErrLayerClosed = fmt.Errorf("layer closed")
)

Types

type Descriptor

type Descriptor struct {
	// MediaType describe the type of the content. All text based formats are
	// encoded as utf-8.
	MediaType string `json:"mediaType,omitempty"`

	// Length in bytes of content.
	Length int64 `json:"length,omitempty"`

	// Digest uniquely identifies the content. A byte stream can be verified
	// against against this digest.
	Digest digest.Digest `json:"digest,omitempty"`
}

Descriptor describes targeted content. Used in conjunction with a blob store, a descriptor can be used to fetch, store and target any kind of blob. The struct also describes the wire protocol format. Fields should only be added but never changed.

type ErrLayerInvalidDigest

type ErrLayerInvalidDigest struct {
	Digest digest.Digest
	Reason error
}

ErrLayerInvalidDigest returned when tarsum check fails.

func (ErrLayerInvalidDigest) Error

func (err ErrLayerInvalidDigest) Error() string

type ErrManifestUnknown

type ErrManifestUnknown struct {
	Name string
	Tag  string
}

ErrManifestUnknown is returned if the manifest is not known by the registry.

func (ErrManifestUnknown) Error

func (err ErrManifestUnknown) Error() string

type ErrManifestUnverified

type ErrManifestUnverified struct{}

ErrManifestUnverified is returned when the registry is unable to verify the manifest.

func (ErrManifestUnverified) Error

func (ErrManifestUnverified) Error() string

type ErrManifestVerification

type ErrManifestVerification []error

ErrManifestVerification provides a type to collect errors encountered during manifest verification. Currently, it accepts errors of all types, but it may be narrowed to those involving manifest verification.

func (ErrManifestVerification) Error

func (errs ErrManifestVerification) Error() string

type ErrRepositoryNameInvalid

type ErrRepositoryNameInvalid struct {
	Name   string
	Reason error
}

ErrRepositoryNameInvalid should be used to denote an invalid repository name. Reason may set, indicating the cause of invalidity.

func (ErrRepositoryNameInvalid) Error

func (err ErrRepositoryNameInvalid) Error() string

type ErrRepositoryUnknown

type ErrRepositoryUnknown struct {
	Name string
}

ErrRepositoryUnknown is returned if the named repository is not known by the registry.

func (ErrRepositoryUnknown) Error

func (err ErrRepositoryUnknown) Error() string

type ErrUnknownLayer

type ErrUnknownLayer struct {
	FSLayer manifest.FSLayer
}

ErrUnknownLayer returned when layer cannot be found.

func (ErrUnknownLayer) Error

func (err ErrUnknownLayer) Error() string

type ErrUnknownManifestRevision

type ErrUnknownManifestRevision struct {
	Name     string
	Revision digest.Digest
}

ErrUnknownManifestRevision is returned when a manifest cannot be found by revision within a repository.

func (ErrUnknownManifestRevision) Error

func (err ErrUnknownManifestRevision) Error() string

type Layer

type Layer interface {
	// http.ServeContent requires an efficient implementation of
	// ReadSeeker.Seek(0, os.SEEK_END).
	io.ReadSeeker
	io.Closer

	// Digest returns the unique digest of the blob.
	Digest() digest.Digest

	// Length returns the length in bytes of the blob.
	Length() int64

	// CreatedAt returns the time this layer was created.
	CreatedAt() time.Time

	// Handler returns an HTTP handler which serves the layer content, whether
	// by providing a redirect directly to the content, or by serving the
	// content itself.
	Handler(r *http.Request) (http.Handler, error)
}

Layer provides a readable and seekable layer object. Typically, implementations are *not* goroutine safe.

type LayerService

type LayerService interface {
	// Exists returns true if the layer exists.
	Exists(digest digest.Digest) (bool, error)

	// Fetch the layer identifed by TarSum.
	Fetch(digest digest.Digest) (Layer, error)

	// Upload begins a layer upload to repository identified by name,
	// returning a handle.
	Upload() (LayerUpload, error)

	// Resume continues an in progress layer upload, returning a handle to the
	// upload. The caller should seek to the latest desired upload location
	// before proceeding.
	Resume(uuid string) (LayerUpload, error)
}

LayerService provides operations on layer files in a backend storage.

type LayerUpload

type LayerUpload interface {
	io.WriteSeeker
	io.ReaderFrom
	io.Closer

	// UUID returns the identifier for this upload.
	UUID() string

	// StartedAt returns the time this layer upload was started.
	StartedAt() time.Time

	// Finish marks the upload as completed, returning a valid handle to the
	// uploaded layer. The digest is validated against the contents of the
	// uploaded layer.
	Finish(digest digest.Digest) (Layer, error)

	// Cancel the layer upload process.
	Cancel() error
}

LayerUpload provides a handle for working with in-progress uploads. Instances can be obtained from the LayerService.Upload and LayerService.Resume.

type ManifestService

type ManifestService interface {
	// Exists returns true if the manifest exists.
	Exists(dgst digest.Digest) (bool, error)

	// Get retrieves the identified by the digest, if it exists.
	Get(dgst digest.Digest) (*manifest.SignedManifest, error)

	// Delete removes the manifest, if it exists.
	Delete(dgst digest.Digest) error

	// Put creates or updates the manifest.
	Put(manifest *manifest.SignedManifest) error

	// Tags lists the tags under the named repository.
	Tags() ([]string, error)

	// ExistsByTag returns true if the manifest exists.
	ExistsByTag(tag string) (bool, error)

	// GetByTag retrieves the named manifest, if it exists.
	GetByTag(tag string) (*manifest.SignedManifest, error)
}

ManifestService provides operations on image manifests.

type Registry

type Registry interface {
	// Repository should return a reference to the named repository. The
	// registry may or may not have the repository but should always return a
	// reference.
	Repository(ctx context.Context, name string) (Repository, error)
}

Registry represents a collection of repositories, addressable by name.

type Repository

type Repository interface {
	// Name returns the name of the repository.
	Name() string

	// Manifests returns a reference to this repository's manifest service.
	Manifests() ManifestService

	// Layers returns a reference to this repository's layers service.
	Layers() LayerService

	// Signatures returns a reference to this repository's signatures service.
	Signatures() SignatureService
}

Repository is a named collection of manifests and layers.

type SignatureService

type SignatureService interface {
	// Get retrieves all of the signature blobs for the specified digest.
	Get(dgst digest.Digest) ([][]byte, error)

	// Put stores the signature for the provided digest.
	Put(dgst digest.Digest, signatures ...[]byte) error
}

SignatureService provides operations on signatures.