manifest

Documentation

Overview

A manifest is a file containing a list of paths and their hash digests, canonically ordered by path in increasing lexicographical order. Manifests are encoded as:

<digest type>:<digest>[SP][SP]<path>[LF]

"shake256" is the only supported digest type. The digest is 64 bytes of hex encoded output of SHAKE256. See golang.org/x/crypto/sha3 and FIPS 202 for details on the SHAKE hash.

Manifest can read and write manifest files. Canonical form is produced when serialized (Manifest.MarshalText). Non-canonical form is a valid manifest and will not produce errors when deserializing.

Interacting with a manifest is typically by path (Manifest.Paths, Manifest.DigestFor) or by a Digest (Manifest.PathsFor).

Blob represents file content and its digest. BlobSet collects related blobs together into a set. NewMemoryBlob provides an in-memory implementation. A manifest, being a file, is also a blob (Manifest.Blob).

Blobs are anonymous files and a manifest gives names to anonymous files. It's possible to view a manifest and its associated blobs as a file system. [NewBucket] creates a storage bucket from a manifest and blob set. NewFromBucket does the inverse: the creation of a manifest and blob set from a storage bucket.

Example

package main

import (
	"context"
	"fmt"

	"github.com/bufbuild/buf/private/pkg/manifest"
	"github.com/bufbuild/buf/private/pkg/storage/storagemem"
)

func main() {
	ctx := context.Background()
	bucket, _ := storagemem.NewReadBucket(
		map[string][]byte{
			"foo": []byte("bar"),
		},
	)
	m, _, _ := manifest.NewFromBucket(ctx, bucket)
	digest, _ := m.DigestFor("foo")
	fmt.Printf("digest[:16]: %s\n", digest.Hex()[:16])
	path, _ := m.PathsFor(digest.String())
	fmt.Printf("path at digest: %s\n", path[0])
}

Output:

digest[:16]: a15163728ed24e1c
path at digest: foo

Index

• func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)
• func NewFromBucket(ctx context.Context, bucket storage.ReadBucket) (*Manifest, *BlobSet, error)
• type Blob
  • func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)
  • func NewMemoryBlobFromReader(content io.Reader) (Blob, error)
  • func NewMemoryBlobFromReaderWithDigester(content io.Reader, digester Digester) (Blob, error)
• type BlobSet
  • func NewBlobSet(ctx context.Context, blobs []Blob, opts ...BlobSetOption) (*BlobSet, error)
  • func (s *BlobSet) BlobFor(digest string) (Blob, bool)
  • func (s *BlobSet) Blobs() []Blob
• type BlobSetOption
  • func BlobSetWithContentValidation() BlobSetOption
  • func BlobSetWithSkipNilBlobs() BlobSetOption
• type Digest
  • func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)
  • func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)
  • func NewDigestFromString(typedDigest string) (*Digest, error)
  • func (d *Digest) Bytes() []byte
  • func (d *Digest) Equal(other Digest) bool
  • func (d *Digest) Hex() string
  • func (d *Digest) String() string
  • func (d *Digest) Type() DigestType
• type DigestType
• type Digester
  • func NewDigester(dtype DigestType) (Digester, error)
• type Manifest
  • func NewFromReader(manifest io.Reader) (*Manifest, error)
  • func (m *Manifest) AddEntry(path string, digest Digest) error
  • func (m *Manifest) Blob() (Blob, error)
  • func (m *Manifest) DigestFor(path string) (*Digest, bool)
  • func (m *Manifest) Digests() []Digest
  • func (m *Manifest) Empty() bool
  • func (m *Manifest) MarshalText() ([]byte, error)
  • func (m *Manifest) Paths() []string
  • func (m *Manifest) PathsFor(digest string) ([]string, bool)
  • func (m *Manifest) Range(f func(path string, digest Digest) error) error
  • func (m *Manifest) UnmarshalText(text []byte) error
• type MemoryBlobOption
  • func MemoryBlobWithDigestValidation() MemoryBlobOption

Examples

• Package

Functions

func BlobEqual

func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)

BlobEqual returns true if blob a is the same as blob b. The digest is checked for equality and the content bytes compared.

An error is returned if an unexpected I/O error occurred when opening, reading, or closing either blob.

func NewFromBucket

func NewFromBucket(
	ctx context.Context,
	bucket storage.ReadBucket,
) (*Manifest, *BlobSet, error)

NewFromBucket creates a manifest and blob set from the bucket's files. Blobs in the blob set use the DigestTypeShake256 digest.

Types

type Blob

type Blob interface {
	Digest() *Digest
	Open(context.Context) (io.ReadCloser, error)
}

Blob is an anonymous file associated with a digest.

func NewMemoryBlob

func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)

NewMemoryBlob takes a digest and a content, and turns it into an in-memory representation of a blob, which returns the digest and an io.ReadCloser for its content.

func NewMemoryBlobFromReader

func NewMemoryBlobFromReader(content io.Reader) (Blob, error)

NewMemoryBlobFromReader creates a memory blob from content, which is read until completion. The returned blob contains all bytes read. If you are using this in a loop, you might better use NewMemoryBlobFromReaderWithDigester so you can reuse your digester.

func NewMemoryBlobFromReaderWithDigester

func NewMemoryBlobFromReaderWithDigester(content io.Reader, digester Digester) (Blob, error)

NewMemoryBlobFromReaderWithDigester creates a memory blob from content with the passed digester. The content is read until completion. The returned blob contains all bytes read.

type BlobSet

type BlobSet struct {
	// contains filtered or unexported fields
}

BlobSet represents a set of deduplicated blobs by their digests.

func NewBlobSet

func NewBlobSet(ctx context.Context, blobs []Blob, opts ...BlobSetOption) (*BlobSet, error)

NewBlobSet receives a slice of blobs, and de-duplicates them into a BlobSet.

func (*BlobSet) BlobFor

func (s *BlobSet) BlobFor(digest string) (Blob, bool)

BlobFor returns the blob for the passed digest string, or nil, ok=false if the digest has no blob in the set.

func (*BlobSet) Blobs

func (s *BlobSet) Blobs() []Blob

Blobs returns a slice of the blobs in the set.

type BlobSetOption

type BlobSetOption func(*blobSetOptions)

BlobSetOption are options passed when creating a new blob set.

func BlobSetWithContentValidation

func BlobSetWithContentValidation() BlobSetOption

BlobSetWithContentValidation turns on content validation for all the blobs when creating a new BlobSet. If this option is on, blobs with the same digest must have the same content (in case blobs with the same digest are sent). If this option is not passed, then the latest duplicated blob digest content will prevail in the set.

func BlobSetWithSkipNilBlobs

func BlobSetWithSkipNilBlobs() BlobSetOption

BlobSetWithSkipNilBlobs allows passing nil blobs in the slice of blobs. The default behavior is that if you pass a nil blob in the slice, you'll get an error from the `NewBlobSet` constructor. If you pass this option, any nil blob will be skipped and the blob set will be built only from the non-nil ones.

type Digest

type Digest struct {
	// contains filtered or unexported fields
}

Digest represents a hash function's value.

func NewDigestFromBytes

func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)

NewDigestFromBytes builds a digest from a type and the digest bytes.

func NewDigestFromHex

func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)

NewDigestFromHex builds a digest from a type and the hexadecimal string of the bytes. It returns an error if the received string is not a valid hex.

func NewDigestFromString

func NewDigestFromString(typedDigest string) (*Digest, error)

NewDigestFromString build a digest from a string representation of it.

func (*Digest) Bytes

func (d *Digest) Bytes() []byte

Bytes returns the digest bytes.

func (*Digest) Equal

func (d *Digest) Equal(other Digest) bool

Equal compares the digest type and bytes with other digest.

func (*Digest) Hex

func (d *Digest) Hex() string

Hex returns the digest bytes in its hexadecimal string representation.

func (*Digest) String

func (d *Digest) String() string

String returns the hash in a manifest's string format: "<type>:<hex>".

func (*Digest) Type

func (d *Digest) Type() DigestType

Type returns the digest type.

type DigestType

type DigestType string

DigestType is the type for digests in this package.

const (
	DigestTypeShake256 DigestType = "shake256"
)

type Digester

type Digester interface {
	Digest(content io.Reader) (*Digest, error)
}

Digester is something that can digest a content into a digest.

func NewDigester

func NewDigester(dtype DigestType) (Digester, error)

NewDigester returns a digester of the requested type.

type Manifest

type Manifest struct {
	// contains filtered or unexported fields
}

Manifest represents a list of paths and their digests.

func NewFromReader

func NewFromReader(manifest io.Reader) (*Manifest, error)

NewFromReader builds a manifest from an encoded manifest, like one produced by Manifest.MarshalText.

func (*Manifest) AddEntry

func (m *Manifest) AddEntry(path string, digest Digest) error

AddEntry adds an entry to the manifest with a path and its digest. It fails if the path already exists in the manifest with a different digest.

func (*Manifest) Blob

func (m *Manifest) Blob() (Blob, error)

Blob returns the manifest as a blob.

func (*Manifest) DigestFor

func (m *Manifest) DigestFor(path string) (*Digest, bool)

DigestFor returns the matching digest for the given path. The path must be an exact match. Digest is nil and ok is false if no digest is found.

func (*Manifest) Digests

func (m *Manifest) Digests() []Digest

Digests returns all unique digests in the manifest. Order is by insertion order.

func (*Manifest) Empty

func (m *Manifest) Empty() bool

Empty returns true if the manifest has no entries.

func (*Manifest) MarshalText

func (m *Manifest) MarshalText() ([]byte, error)

MarshalText encodes the manifest into its canonical form.

func (*Manifest) Paths

func (m *Manifest) Paths() []string

Paths returns all unique paths in the manifest by insertion order.

func (*Manifest) PathsFor

func (m *Manifest) PathsFor(digest string) ([]string, bool)

PathsFor returns one or more matching path for a given digest. The digest is expected to be a lower-case hex encoded value. Returned paths are ordered by insertion time. Paths is nil and ok is false if no paths are found.

func (*Manifest) Range

func (m *Manifest) Range(f func(path string, digest Digest) error) error

Range invokes a function for all the paths in the manifest, passing the path and its digest. Paths are invoked by insertion order. This func will stop iterating if an error is returned.

func (*Manifest) UnmarshalText

func (m *Manifest) UnmarshalText(text []byte) error

UnmarshalText decodes a manifest from text.

See NewFromReader if your manifest is available in an io.Reader.

type MemoryBlobOption

type MemoryBlobOption func(*memoryBlobOptions)

MemoryBlobOption are options passed when creating a new memory blob.

func MemoryBlobWithDigestValidation

func MemoryBlobWithDigestValidation() MemoryBlobOption

MemoryBlobWithDigestValidation checks that the passed content and digest match.