manifest

Documentation

Overview

Package manifest defines generic content-addressable storage APIs which are used to store and verify Buf modules. The primary data types relevant to consumers of this package are Manifest, Blob, BlobSet, and Digest.

Manifest can read and write manifest files. The canonical form of a manifest is produced when serialized with Manifest.MarshalText. Interacting with a manifest is typically by path (Manifest.Paths, Manifest.DigestFor) or by a Digest (Manifest.PathsFor). Note that it is possible for multiple paths in the manifest to have to same digest (and content), however a given path only has one digest.

Blob represents file content and its digest. NewMemoryBlob provides an in-memory implementation of a Blob. A manifest, being a file, is also a blob. The Manifest.Blob function converts a manifest to a Blob.

BlobSet collects related blobs together into a unique set (de-duplicating blobs with the same digest and content).

Blobs are anonymous files and a manifest gives names to anonymous files. It's possible to view a manifest and its associated blobs as a file system. NewFromBucket creates a manifest and blob set from a storage bucket.

Aside from the Manifest.MarshalText encoding, this package does not define the representation of its data types on the wire. See the private/bufpkg/bufmanifest package and buf/alpha/registry/v1alpha1/module.proto for details on how these data types are represented in Protobuf.

Example

package main

import (
	"context"
	"fmt"

	"github.com/bufbuild/buf/private/pkg/manifest"
	"github.com/bufbuild/buf/private/pkg/storage/storagemem"
)

func main() {
	ctx := context.Background()
	bucket, _ := storagemem.NewReadBucket(
		map[string][]byte{
			"foo": []byte("bar"),
		},
	)
	m, _, _ := manifest.NewFromBucket(ctx, bucket)
	encoded, _ := m.MarshalText()
	fmt.Println("Canonical manifest encoding:")
	fmt.Println(string(encoded))
	digest, _ := m.DigestFor("foo")
	fmt.Printf("digest[:16]: %s\n", digest.Hex()[:16])
	paths, _ := m.PathsFor(digest.String())
	fmt.Printf("paths for digest: %v\n", paths)
}

Output:

Canonical manifest encoding:
shake256:a15163728ed24e1c7eddc03e92cb421940a0d1fe653bee3294358adf391d4da26adc110656392ff324f01dcdc9596196f1e66a3a071bb226e71a6f53ca2bf4ad  foo

digest[:16]: a15163728ed24e1c
paths for digest: [foo]

Index

• func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)
• func NewFromBucket(ctx context.Context, bucket storage.ReadBucket) (*Manifest, *BlobSet, error)
• type Blob
  • func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)
  • func NewMemoryBlobFromReader(content io.Reader) (Blob, error)
  • func NewMemoryBlobFromReaderWithDigester(content io.Reader, digester Digester) (Blob, error)
• type BlobSet
  • func NewBlobSet(ctx context.Context, blobs []Blob) (*BlobSet, error)
  • func (s *BlobSet) BlobFor(digest string) (Blob, bool)
  • func (s *BlobSet) Blobs() []Blob
• type Digest
  • func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)
  • func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)
  • func NewDigestFromString(typedDigest string) (*Digest, error)
  • func (d *Digest) Bytes() []byte
  • func (d *Digest) Equal(other Digest) bool
  • func (d *Digest) Hex() string
  • func (d *Digest) String() string
  • func (d *Digest) Type() DigestType
• type DigestType
• type Digester
  • func NewDigester(dtype DigestType) (Digester, error)
• type Manifest
  • func NewFromReader(manifest io.Reader) (*Manifest, error)
  • func (m *Manifest) AddEntry(path string, digest Digest) error
  • func (m *Manifest) Blob() (Blob, error)
  • func (m *Manifest) DigestFor(path string) (*Digest, bool)
  • func (m *Manifest) Digests() []Digest
  • func (m *Manifest) Empty() bool
  • func (m *Manifest) MarshalText() ([]byte, error)
  • func (m *Manifest) Paths() []string
  • func (m *Manifest) PathsFor(digest string) ([]string, bool)
  • func (m *Manifest) Range(f func(path string, digest Digest) error) error
  • func (m *Manifest) UnmarshalText(text []byte) error
• type MemoryBlobOption
  • func MemoryBlobWithDigestValidation() MemoryBlobOption

Examples

• Package

Functions

func BlobEqual

func BlobEqual(ctx context.Context, a, b Blob) (_ bool, retErr error)

BlobEqual returns true if blob a is the same as blob b. The digest is checked for equality and the content bytes compared.

An error is returned if an unexpected I/O error occurred when opening, reading, or closing either blob.

func NewFromBucket

func NewFromBucket(
	ctx context.Context,
	bucket storage.ReadBucket,
) (*Manifest, *BlobSet, error)

NewFromBucket creates a manifest and blob set from the bucket's files. Blobs in the blob set use the DigestTypeShake256 digest.

Types

type Blob

type Blob interface {
	// Digest returns the digest of the blob's content.
	Digest() *Digest
	// Open returns an io.ReadCloser of the blob's content.
	// Callers should always ensure this is closed.
	Open(context.Context) (io.ReadCloser, error)
}

Blob is an anonymous file and a digest of its contents.

func NewMemoryBlob

func NewMemoryBlob(digest Digest, content []byte, opts ...MemoryBlobOption) (Blob, error)

NewMemoryBlob takes a digest and a content, and turns it into an in-memory representation of a blob, which returns the digest and an io.ReadCloser for its content.

func NewMemoryBlobFromReader

func NewMemoryBlobFromReader(content io.Reader) (Blob, error)

NewMemoryBlobFromReader creates a memory blob from content, which is read until completion. The returned blob contains all bytes read. If you are using this in a loop, it is recommended to use NewMemoryBlobFromReaderWithDigester to reuse your digester.

func NewMemoryBlobFromReaderWithDigester

func NewMemoryBlobFromReaderWithDigester(content io.Reader, digester Digester) (Blob, error)

NewMemoryBlobFromReaderWithDigester creates a memory blob from content with the passed digester. The content is read until completion. The returned blob contains all bytes read.

type BlobSet

type BlobSet struct {
	// contains filtered or unexported fields
}

BlobSet represents a set of deduplicated blobs by their digests.

func NewBlobSet

func NewBlobSet(ctx context.Context, blobs []Blob) (*BlobSet, error)

NewBlobSet receives a slice of blobs, and de-duplicates them into a BlobSet.

func (*BlobSet) BlobFor

func (s *BlobSet) BlobFor(digest string) (Blob, bool)

BlobFor returns the blob for the passed digest string, or the zero value (Blob, false) if the digest has no blob in the set.

func (*BlobSet) Blobs

func (s *BlobSet) Blobs() []Blob

Blobs returns a slice of the blobs in the set, in unspecified order.

type Digest

type Digest struct {
	// contains filtered or unexported fields
}

Digest represents a hash function's value.

func NewDigestFromBytes

func NewDigestFromBytes(dtype DigestType, digest []byte) (*Digest, error)

NewDigestFromBytes builds a digest from a type and the digest bytes.

func NewDigestFromHex

func NewDigestFromHex(dtype DigestType, hexstr string) (*Digest, error)

NewDigestFromHex builds a digest from a type and the hexadecimal string of the bytes. It returns an error if the received string is not a valid hex string.

func NewDigestFromString

func NewDigestFromString(typedDigest string) (*Digest, error)

NewDigestFromString build a digest from a string representation of it. See Digest.String for details on the string representation.

func (*Digest) Bytes

func (d *Digest) Bytes() []byte

Bytes returns the digest bytes.

func (*Digest) Equal

func (d *Digest) Equal(other Digest) bool

Equal compares the digest type and bytes with other digest.

func (*Digest) Hex

func (d *Digest) Hex() string

Hex returns the digest bytes in its hexadecimal string representation.

func (*Digest) String

func (d *Digest) String() string

String returns the hash in a manifest's string format:

<type>:<hex>

func (*Digest) Type

func (d *Digest) Type() DigestType

Type returns the digest type.

type DigestType

type DigestType string

DigestType specifies supported digest types.

const (
	// DigestTypeShake256 is the digest type for the SHAKE256 digest.
	DigestTypeShake256 DigestType = "shake256"
)

type Digester

type Digester interface {
	// Digest returns the calculated digest for the specified content.
	Digest(content io.Reader) (*Digest, error)
}

Digester is something that can calculate a digest for content.

func NewDigester

func NewDigester(dtype DigestType) (Digester, error)

NewDigester returns a digester of the requested type.

type Manifest

type Manifest struct {
	// contains filtered or unexported fields
}

Manifest represents a list of paths and their digests.

func NewFromReader

func NewFromReader(manifest io.Reader) (*Manifest, error)

NewFromReader builds a manifest from an encoded manifest, like one produced by Manifest.MarshalText.

func (*Manifest) AddEntry

func (m *Manifest) AddEntry(path string, digest Digest) error

AddEntry adds an entry to the manifest with a path and its digest. It fails if the path already exists in the manifest with a different digest.

func (*Manifest) Blob

func (m *Manifest) Blob() (Blob, error)

Blob returns the manifest as a blob. The Blob content is set to the canonical representation of a manifest (Manifest.MarshalText), and the digest is set to the SHAKE256 digest of the content.

func (*Manifest) DigestFor

func (m *Manifest) DigestFor(path string) (*Digest, bool)

DigestFor returns the matching digest for the given path. The path must be an exact match. Returns (nil, false) if no digest is found.

func (*Manifest) Digests

func (m *Manifest) Digests() []Digest

Digests returns all unique digests in the manifest, in insertion order.

func (*Manifest) Empty

func (m *Manifest) Empty() bool

Empty returns true if the manifest has no entries.

func (*Manifest) MarshalText

func (m *Manifest) MarshalText() ([]byte, error)

MarshalText encodes the manifest into its canonical form, consisting of an ordered list of paths and their hash digests. Manifests are encoded as:

<digest_type>:<digest>[SP][SP]<path>[LF]

The only supported digest_type is shake256. The digest is 64 bytes of hex encoded output of SHAKE256. See golang.org/x/crypto/sha3 and FIPS 202 for details on the SHAKE hash.

An example encoded manifest for the acme/petapis module is:

shake256:cd22db48cf7c274bbffcb5494a854000cd21b074df7c6edabbd0102c4be8d7623e3931560fcda7acfab286ae1d4f506911daa31f223ee159f59ffce0c7acbbaa  buf.lock
shake256:3b353aa5aacd11015e8577f16e2c4e7a242ce773d8e3a16806795bb94f76e601b0db9bf42d5e1907fda63303e1fa1c65f1c175ecc025a3ef29c3456ad237ad84  buf.md
shake256:7c88a20cf931702d042a4ddee3fde5de84814544411f1c62dbf435b1b81a12a8866a070baabcf8b5a0d31675af361ccb2d93ddada4cdcc11bab7ea3d8d7c4667  buf.yaml
shake256:9db25155eafd19b36882cff129daac575baa67ee44d1cb1fd3894342b28c72b83eb21aa595b806e9cb5344759bc8308200c5af98e4329aa83014dde99afa903a  pet/v1/pet.proto

func (*Manifest) Paths

func (m *Manifest) Paths() []string

Paths returns all unique paths in the manifest by insertion order.

func (*Manifest) PathsFor

func (m *Manifest) PathsFor(digest string) ([]string, bool)

PathsFor returns one or more matching path for a given digest. The digest is expected to be a lower-case hex encoded value. Returned paths are ordered by insertion time. Returns (nil, false) if no paths are found.

func (*Manifest) Range

func (m *Manifest) Range(f func(path string, digest Digest) error) error

Range invokes a function for all the paths in the manifest, passing the path and its digest. Paths are invoked by insertion order. This func will stop iterating if an error is returned.

func (*Manifest) UnmarshalText

func (m *Manifest) UnmarshalText(text []byte) error

UnmarshalText decodes a manifest from text. See Manifest.MarshalText for the expected encoding of a manifest.

See NewFromReader if your manifest is available in an io.Reader.

type MemoryBlobOption

type MemoryBlobOption func(*memoryBlobOptions)

MemoryBlobOption are options passed when creating a new memory blob.

func MemoryBlobWithDigestValidation

func MemoryBlobWithDigestValidation() MemoryBlobOption

MemoryBlobWithDigestValidation checks that the passed content and digest match.