manifest

Documentation

Overview

A manifest is a file containing a list of paths and their hash digests, canonically ordered by path in increasing lexicographical order. Manifests are encoded as:

<digest type>:<digest>[SP][SP]<path>[LF]

"shake256" is the only supported digest type. The digest is 64 bytes of hex encoded output of SHAKE256. See golang.org/x/crypto/sha3 and FIPS 202 for details on the SHAKE hash.

Manifest can read and write manifest files. Canonical form is produced when serialized (Manifest.MarshalText). Non-canonical form is a valid manifest and will not produce errors when deserializing.

Interacting with a manifest is typically by path (Manifest.Paths, Manifest.DigestFor) or by a Digest (Manifest.PathsFor).

Blob represents file content and its digest. BlobSet collects related blobs together into a set. NewMemoryBlob provides an in-memory implementation. A manifest, being a file, is also a blob (Manifest.Blob).

Blobs are anonymous files and a manifest gives names to anonymous files. It's possible to view a manifest and its associated blobs as a file system. NewBucket creates a storage bucket from a manifest and blob set. NewFromBucket does the inverse: the creation of a manifest and blob set from a storage bucket.

Example

package main

import (
	"context"
	"fmt"

	"github.com/xiaoyawei/buf/private/pkg/manifest"
	"github.com/xiaoyawei/buf/private/pkg/storage/storagemem"
)

func main() {
	ctx := context.Background()
	bucket, _ := storagemem.NewReadBucket(
		map[string][]byte{
			"foo": []byte("bar"),
		},
	)
	m, _, _ := manifest.NewFromBucket(ctx, bucket)
	digest, _ := m.DigestFor("foo")
	fmt.Printf("digest[:16]: %s\n", digest.Hex()[:16])
	path, _ := m.PathsFor(digest.String())
	fmt.Printf("path at digest: %s\n", path[0])
}

Output:

digest[:16]: a15163728ed24e1c
path at digest: foo

Index

• func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)
• func NewBucket(m Manifest, blobs BlobSet, opts ...BucketOption) (storage.ReadBucket, error)
• func NewFromBucket(ctx context.Context, bucket storage.ReadBucket) (*Manifest, *BlobSet, error)
• type Blob
  • func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)
  • func NewMemoryBlobFromReader(content io.Reader) (Blob, error)
  • func NewMemoryBlobFromReaderWithDigester(content io.Reader, digester Digester) (Blob, error)
• type BlobSet
  • func NewBlobSet(ctx context.Context, blobs []Blob, opts ...BlobSetOption) (*BlobSet, error)
  • func (s *BlobSet) BlobFor(digest string) (Blob, bool)
  • func (s *BlobSet) Blobs() []Blob
• type BlobSetOption
  • func BlobSetWithContentValidation() BlobSetOption
• type BucketOption
  • func BucketWithAllManifestBlobsValidation() BucketOption
  • func BucketWithNoExtraBlobsValidation() BucketOption
• type Digest
  • func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)
  • func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)
  • func NewDigestFromString(typedDigest string) (*Digest, error)
  • func (d *Digest) Bytes() []byte
  • func (d *Digest) Equal(other Digest) bool
  • func (d *Digest) Hex() string
  • func (d *Digest) String() string
  • func (d *Digest) Type() DigestType
• type DigestType
• type Digester
  • func NewDigester(dtype DigestType) (Digester, error)
• type Manifest
  • func New() *Manifest
  • func NewFromReader(manifest io.Reader) (*Manifest, error)
  • func (m *Manifest) AddEntry(path string, digest Digest) error
  • func (m *Manifest) Blob() (Blob, error)
  • func (m *Manifest) DigestFor(path string) (*Digest, bool)
  • func (m *Manifest) MarshalText() ([]byte, error)
  • func (m *Manifest) Paths() []string
  • func (m *Manifest) PathsFor(digest string) ([]string, bool)
  • func (m *Manifest) UnmarshalText(text []byte) error
• type MemoryBlobOption
  • func MemoryBlobWithDigestValidation() MemoryBlobOption

Examples

• Package

Functions

func BlobEqual

func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)

BlobEqual returns true if blob a is the same as blob b. The digest is checked for equality and the content bytes compared.

An error is returned if an unexpected I/O error occurred when opening, reading, or closing either blob.

func NewBucket

func NewBucket(m Manifest, blobs BlobSet, opts ...BucketOption) (storage.ReadBucket, error)

NewBucket takes a manifest and a blob set and builds a readable storage bucket that contains the files in the manifest.

func NewFromBucket

func NewFromBucket(
	ctx context.Context,
	bucket storage.ReadBucket,
) (*Manifest, *BlobSet, error)

NewFromBucket creates a manifest and blob set from the bucket's files. Blobs in the blob set use the DigestTypeShake256 digest.

Types

type Blob

type Blob interface {
	Digest() *Digest
	Open(context.Context) (io.ReadCloser, error)
}

Blob is an anonymous file associated with a digest.

func NewMemoryBlob

func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)

NewMemoryBlob takes a digest and a content, and turns it into an in-memory representation of a blob, which returns the digest and an io.ReadCloser for its content.

func NewMemoryBlobFromReader

func NewMemoryBlobFromReader(content io.Reader) (Blob, error)

NewMemoryBlobFromReader creates a memory blob from content, which is read until completion. The returned blob contains all bytes read. If you are using this in a loop, you might better use NewMemoryBlobFromReaderWithDigester so you can reuse your digester.

func NewMemoryBlobFromReaderWithDigester

func NewMemoryBlobFromReaderWithDigester(content io.Reader, digester Digester) (Blob, error)

NewMemoryBlobFromReaderWithDigester creates a memory blob from content with the passed digester. The content is read until completion. The returned blob contains all bytes read.

type BlobSet

type BlobSet struct {
	// contains filtered or unexported fields
}

BlobSet represents a set of deduplicated blobs by their digests.

func NewBlobSet

func NewBlobSet(ctx context.Context, blobs []Blob, opts ...BlobSetOption) (*BlobSet, error)

NewBlobSet receives a slice of blobs, and de-duplicates them into a BlobSet.

func (*BlobSet) BlobFor

func (s *BlobSet) BlobFor(digest string) (Blob, bool)

BlobFor returns the blob for the passed digest string, or nil, ok=false if the digest has no blob in the set.

func (*BlobSet) Blobs

func (s *BlobSet) Blobs() []Blob

Blobs returns a slice of the blobs in the set.

type BlobSetOption

type BlobSetOption func(*blobSetOptions)

BlobSetOption are options passed when creating a new blob set.

func BlobSetWithContentValidation

func BlobSetWithContentValidation() BlobSetOption

BlobSetWithContentValidation turns on content validation for all the blobs when creating a new BlobSet. If this option is on, blobs with the same digest must have the same content (in case blobs with the same digest are sent). If this option is not passed, then the latest duplicated blob digest content will prevail in the set.

type BucketOption

type BucketOption func(*bucketOptions)

BucketOption are options passed when creating a new manifest bucket.

func BucketWithAllManifestBlobsValidation

func BucketWithAllManifestBlobsValidation() BucketOption

BucketWithAllManifestBlobsValidation validates that all manifest digests have a corresponding blob in the blob set. If this option is not passed, then buckets with partial/incomplete blobs are allowed.

func BucketWithNoExtraBlobsValidation

func BucketWithNoExtraBlobsValidation() BucketOption

BucketWithNoExtraBlobsValidation validates that the passed blob set has no additional blobs beyond the ones in the manifest.

type Digest

type Digest struct {
	// contains filtered or unexported fields
}

Digest represents a hash function's value.

func NewDigestFromBytes

func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)

NewDigestFromBytes builds a digest from a type and the digest bytes.

func NewDigestFromHex

func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)

NewDigestFromHex builds a digest from a type and the hexadecimal string of the bytes. It returns an error if the received string is not a valid hex.

func NewDigestFromString

func NewDigestFromString(typedDigest string) (*Digest, error)

NewDigestFromString build a digest from a string representation of it.

func (*Digest) Bytes

func (d *Digest) Bytes() []byte

Bytes returns the digest bytes.

func (*Digest) Equal

func (d *Digest) Equal(other Digest) bool

Equal compares the digest type and bytes with other digest.

func (*Digest) Hex

func (d *Digest) Hex() string

Hex returns the digest bytes in its hexadecimal string representation.

func (*Digest) String

func (d *Digest) String() string

String returns the hash in a manifest's string format: "<type>:<hex>".

func (*Digest) Type

func (d *Digest) Type() DigestType

Type returns the digest type.

type DigestType

type DigestType string

DigestType is the type for digests in this package.

const (
	DigestTypeShake256 DigestType = "shake256"
)

type Digester

type Digester interface {
	Digest(content io.Reader) (*Digest, error)
}

Digester is something that can digest a content into a digest.

func NewDigester

func NewDigester(dtype DigestType) (Digester, error)

NewDigester returns a digester of the requested type.

type Manifest

type Manifest struct {
	// contains filtered or unexported fields
}

Manifest represents a list of paths and their digests.

func New

func New() *Manifest

New creates an empty manifest.

func NewFromReader

func NewFromReader(manifest io.Reader) (*Manifest, error)

NewFromReader builds a manifest from an encoded manifest, like one produced by Manifest.MarshalText.

func (*Manifest) AddEntry

func (m *Manifest) AddEntry(path string, digest Digest) error

AddEntry adds an entry to the manifest with a path and its digest. It fails if the path already exists in the manifest with a different digest.

func (*Manifest) Blob

func (m *Manifest) Blob() (Blob, error)

Blob returns the manifest as a blob.

func (*Manifest) DigestFor

func (m *Manifest) DigestFor(path string) (*Digest, bool)

DigestFor returns the matching digest for the given path. The path must be an exact match. Digest is nil and ok is false if no digest is found.

func (*Manifest) MarshalText

func (m *Manifest) MarshalText() ([]byte, error)

MarshalText encodes the manifest into its canonical form.

func (*Manifest) Paths

func (m *Manifest) Paths() []string

Paths returns all paths in the manifest.

func (*Manifest) PathsFor

func (m *Manifest) PathsFor(digest string) ([]string, bool)

PathsFor returns one or more matching path for a given digest. The digest is expected to be a lower-case hex encoded value. Returned paths are unordered. Paths is nil and ok is false if no paths are found.

func (*Manifest) UnmarshalText

func (m *Manifest) UnmarshalText(text []byte) error

UnmarshalText decodes a manifest from text.

See NewFromReader if your manifest is available in an io.Reader.

type MemoryBlobOption

type MemoryBlobOption func(*memoryBlobOptions)

MemoryBlobOption are options passed when creating a new memory blob.

func MemoryBlobWithDigestValidation

func MemoryBlobWithDigestValidation() MemoryBlobOption

MemoryBlobWithDigestValidation checks that the passed content and digest match.