digest

README

go-digest

Common digest package used across the container ecosystem.

Please see the godoc for more information.

What is a digest?

A digest is just a hash.

The most common use case for a digest is to create a content identifier for use in Content Addressable Storage systems:

id := digest.FromBytes([]byte("my content"))

In the example above, the id can be used to uniquely identify the byte slice "my content". This allows two disparate applications to agree on a verifiable identifier without having to trust one another.

An identifying digest can be verified, as follows:

if id != digest.FromBytes([]byte("my content")) {
  return errors.New("the content has changed!")
}

A Verifier type can be used to handle cases where an io.Reader makes more sense:

rd := getContent()
verifier := id.Verifier()
io.Copy(verifier, rd)

if !verifier.Verified() {
  return errors.New("the content has changed!")
}

Using Merkle DAGs, this can power a rich, safe, content distribution system.

Usage

While the godoc is considered the best resource, a few important items need to be called out when using this package.

1. Make sure to import the hash implementations into your application or the package will panic. You should have something like the following in the main (or other entrypoint) of your application:

   import (
       _ "crypto/sha256"
       _ "crypto/sha512"
   )
   

   This may seem inconvenient but it allows you replace the hash implementations with others, such as https://github.com/stevvooe/resumable.

2. Even though digest.Digest may be assemblable as a string, always verify your input with digest.Parse or use Digest.Validate when accepting untrusted input. While there are measures to avoid common problems, this will ensure you have valid digests in the rest of your application.

3. While alternative encodings of hash values (digests) are possible (for example, base64), this package deals exclusively with hex-encoded digests.

Stability

The Go API, at this stage, is considered stable, unless otherwise noted.

As always, before using a package export, read the godoc.

Contributing

This package is considered fairly complete. It has been in production in thousands (millions?) of deployments and is fairly battle-hardened. New additions will be met with skepticism. If you think there is a missing feature, please file a bug clearly describing the problem and the alternatives you tried before submitting a PR.

Code of Conduct

Participation in the OpenContainers community is governed by OpenContainer's Code of Conduct.

Security

If you find an issue, please follow the security protocol to report it.

Copyright and license

Copyright © 2019, 2020 OCI Contributors Copyright © 2016 Docker, Inc. All rights reserved, except as follows. Code is released under the Apache 2.0 license. This README.md file and the CONTRIBUTING.md file are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file LICENSE.docs. You may obtain a duplicate copy of the same license, titled CC BY-SA 4.0, at http://creativecommons.org/licenses/by-sa/4.0/.

Documentation

Overview

Package digest provides a generalized type to opaquely represent message digests and their operations within the registry. The Digest type is designed to serve as a flexible identifier in a content-addressable system. More importantly, it provides tools and wrappers to work with hash.Hash-based digests with little effort.

Basics

The format of a digest is simply a string with two parts, dubbed the "algorithm" and the "digest", separated by a colon:

<algorithm>:<digest>

An example of a sha256 digest representation follows:

sha256:7173b809ca12ec5dee4506cd86be934c4596dd234ee82c0662eac04a8c2c71dc

The "algorithm" portion defines both the hashing algorithm used to calculate the digest and the encoding of the resulting digest, which defaults to "hex" if not otherwise specified. Currently, all supported algorithms have their digests encoded in hex strings.

In the example above, the string "sha256" is the algorithm and the hex bytes are the "digest".

Because the Digest type is simply a string, once a valid Digest is obtained, comparisons are cheap, quick and simple to express with the standard equality operator.

Verification

The main benefit of using the Digest type is simple verification against a given digest. The Verifier interface, modeled after the stdlib hash.Hash interface, provides a common write sink for digest verification. After writing is complete, calling the Verifier.Verified method will indicate whether or not the stream of bytes matches the target digest.

Missing Features

In addition to the above, we intend to add the following features to this package:

1. A Digester type that supports write sink digest calculation.

2. Suspend and resume of ongoing digest calculations to support efficient digest verification in the registry.

Index

• Variables
• type Algorithm
  • func (a Algorithm) Available() bool
  • func (a Algorithm) Digester() Digester
  • func (a Algorithm) Encode(d []byte) string
  • func (a Algorithm) FromBytes(p []byte) Digest
  • func (a Algorithm) FromReader(rd io.Reader) (Digest, error)
  • func (a Algorithm) FromString(s string) Digest
  • func (a Algorithm) Hash() hash.Hash
  • func (a *Algorithm) Set(value string) error
  • func (a Algorithm) Size() int
  • func (a Algorithm) String() string
  • func (a Algorithm) Validate(encoded string) error
• type Digest
  • func FromBytes(p []byte) Digest
  • func FromReader(rd io.Reader) (Digest, error)
  • func FromString(s string) Digest
  • func NewDigest(alg Algorithm, h hash.Hash) Digest
  • func NewDigestFromBytes(alg Algorithm, p []byte) Digest
  • func NewDigestFromEncoded(alg Algorithm, encoded string) Digest
  • func NewDigestFromHex(alg, hex string) Digest
  • func Parse(s string) (Digest, error)
  • func (d Digest) Algorithm() Algorithm
  • func (d Digest) Encoded() string
  • func (d Digest) Hex() string
  • func (d Digest) String() string
  • func (d Digest) Validate() error
  • func (d Digest) Verifier() Verifier
• type Digester
• type Verifier

Variables

var (
	// ErrDigestInvalidFormat returned when digest format invalid.
	ErrDigestInvalidFormat = fmt.Errorf("invalid checksum digest format")

	// ErrDigestInvalidLength returned when digest has invalid length.
	ErrDigestInvalidLength = fmt.Errorf("invalid checksum digest length")

	// ErrDigestUnsupported returned when the digest algorithm is unsupported.
	ErrDigestUnsupported = fmt.Errorf("unsupported digest algorithm")
)

var DigestRegexp = regexp.MustCompile(`[a-z0-9]+(?:[.+_-][a-z0-9]+)*:[a-zA-Z0-9=_-]+`)

DigestRegexp matches valid digest types.

var DigestRegexpAnchored = regexp.MustCompile(`^` + DigestRegexp.String() + `$`)

DigestRegexpAnchored matches valid digest types, anchored to the start and end of the match.

Types

type Algorithm

type Algorithm string

Algorithm identifies and implementation of a digester by an identifier. Note the that this defines both the hash algorithm used and the string encoding.

const (
	SHA256 Algorithm = "sha256" // sha256 with hex encoding (lower case only)
	SHA384 Algorithm = "sha384" // sha384 with hex encoding (lower case only)
	SHA512 Algorithm = "sha512" // sha512 with hex encoding (lower case only)

	// Canonical is the primary digest algorithm used with the distribution
	// project. Other digests may be used but this one is the primary storage
	// digest.
	Canonical = SHA256
)

supported digest types

func (Algorithm) Available

func (a Algorithm) Available() bool

Available returns true if the digest type is available for use. If this returns false, Digester and Hash will return nil.

func (Algorithm) Digester

func (a Algorithm) Digester() Digester

Digester returns a new digester for the specified algorithm. If the algorithm does not have a digester implementation, nil will be returned. This can be checked by calling Available before calling Digester.

func (Algorithm) Encode

func (a Algorithm) Encode(d []byte) string

Encode encodes the raw bytes of a digest, typically from a hash.Hash, into the encoded portion of the digest.

func (Algorithm) FromBytes

func (a Algorithm) FromBytes(p []byte) Digest

FromBytes digests the input and returns a Digest.

func (Algorithm) FromReader

func (a Algorithm) FromReader(rd io.Reader) (Digest, error)

FromReader returns the digest of the reader using the algorithm.

func (Algorithm) FromString

func (a Algorithm) FromString(s string) Digest

FromString digests the string input and returns a Digest.

func (Algorithm) Hash

func (a Algorithm) Hash() hash.Hash

Hash returns a new hash as used by the algorithm. If not available, the method will panic. Check Algorithm.Available() before calling.

func (*Algorithm) Set

func (a *Algorithm) Set(value string) error

Set implemented to allow use of Algorithm as a command line flag.

func (Algorithm) Size

func (a Algorithm) Size() int

Size returns number of bytes returned by the hash.

func (Algorithm) String

func (a Algorithm) String() string

func (Algorithm) Validate

func (a Algorithm) Validate(encoded string) error

Validate validates the encoded portion string

type Digest

type Digest string

Digest allows simple protection of hex formatted digest strings, prefixed by their algorithm. Strings of type Digest have some guarantee of being in the correct format and it provides quick access to the components of a digest string.

The following is an example of the contents of Digest types:

sha256:7173b809ca12ec5dee4506cd86be934c4596dd234ee82c0662eac04a8c2c71dc

This allows to abstract the digest behind this type and work only in those terms.

func FromBytes

func FromBytes(p []byte) Digest

FromBytes digests the input and returns a Digest.

func FromReader

func FromReader(rd io.Reader) (Digest, error)

FromReader consumes the content of rd until io.EOF, returning canonical digest.

func FromString

func FromString(s string) Digest

FromString digests the input and returns a Digest.

func NewDigest

func NewDigest(alg Algorithm, h hash.Hash) Digest

NewDigest returns a Digest from alg and a hash.Hash object.

func NewDigestFromBytes

func NewDigestFromBytes(alg Algorithm, p []byte) Digest

NewDigestFromBytes returns a new digest from the byte contents of p. Typically, this can come from hash.Hash.Sum(...) or xxx.SumXXX(...) functions. This is also useful for rebuilding digests from binary serializations.

func NewDigestFromEncoded

func NewDigestFromEncoded(alg Algorithm, encoded string) Digest

NewDigestFromEncoded returns a Digest from alg and the encoded digest.

func NewDigestFromHex

func NewDigestFromHex(alg, hex string) Digest

NewDigestFromHex is deprecated. Please use NewDigestFromEncoded.

func Parse

func Parse(s string) (Digest, error)

Parse parses s and returns the validated digest object. An error will be returned if the format is invalid.

func (Digest) Algorithm

func (d Digest) Algorithm() Algorithm

Algorithm returns the algorithm portion of the digest. This will panic if the underlying digest is not in a valid format.

func (Digest) Encoded

func (d Digest) Encoded() string

Encoded returns the encoded portion of the digest. This will panic if the underlying digest is not in a valid format.

func (Digest) Hex

func (d Digest) Hex() string

Hex is deprecated. Please use Digest.Encoded.

func (Digest) String

func (d Digest) String() string

func (Digest) Validate

func (d Digest) Validate() error

Validate checks that the contents of d is a valid digest, returning an error if not.

func (Digest) Verifier

func (d Digest) Verifier() Verifier

Verifier returns a writer object that can be used to verify a stream of content against the digest. If the digest is invalid, the method will panic.

type Digester

type Digester interface {
	Hash() hash.Hash // provides direct access to underlying hash instance.
	Digest() Digest
}

Digester calculates the digest of written data. Writes should go directly to the return value of Hash, while calling Digest will return the current value of the digest.

type Verifier

type Verifier interface {
	io.Writer

	// Verified will return true if the content written to Verifier matches
	// the digest.
	Verified() bool
}

Verifier presents a general verification interface to be used with message digests and other byte stream verifications. Users instantiate a Verifier from one of the various methods, write the data under test to it then check the result with the Verified method.