digest

package
v0.0.0-...-07366c6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 23, 2025 License: Apache-2.0 Imports: 25 Imported by: 38

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var EmptySet = Set{}

EmptySet is an instance of Set that contains zero elements.

View Source 
var SupportedDigestFunctions = []remoteexecution.DigestFunction_Value{
	remoteexecution.DigestFunction_MD5,
	remoteexecution.DigestFunction_SHA1,
	remoteexecution.DigestFunction_SHA256,
	remoteexecution.DigestFunction_SHA256TREE,
	remoteexecution.DigestFunction_SHA384,
	remoteexecution.DigestFunction_SHA512,
}

SupportedDigestFunctions is the list of digest functions supported by digest.Digest, using the enumeration values that are part of the Remote Execution protocol.

Functions

func RemoveUnsupportedDigestFunctions 

func RemoveUnsupportedDigestFunctions(reported []remoteexecution.DigestFunction_Value) []remoteexecution.DigestFunction_Value

RemoveUnsupportedDigestFunctions returns the intersection between a list of provided digest functions and ones supported by this implementation. Results are guaranteed to be deduplicated and in alphabetic order.

Types

type Digest 

type Digest struct {
	// contains filtered or unexported fields
}

Digest holds the identification of an object stored in the Content Addressable Storage (CAS) or Action Cache (AC). The use of this object is preferred over remoteexecution.Digest for a couple of reasons.

  • Instances of these objects are guaranteed not to contain any degenerate values. The hash has already been decoded from hexadecimal to binary. The size is non-negative.
  • They keep track of the instance as part of the digest, which allows us to keep function signatures across the codebase simple.
  • They provide utility functions for deriving new digests from them. This ensures that outputs of build actions automatically use the same instance name and hashing algorithm.

Because Digest objects are frequently used as keys (as part of caching data structures or to construct sets without duplicate values), this implementation immediately constructs a key representation upon creation. All functions that extract individual components (e.g., GetInstanceName(), GetHash*() and GetSizeBytes()) operate directly on the key format. 

var BadDigest Digest

BadDigest is a default instance of Digest. It can, for example, be used as a function return value for error cases.

func MustNewDigest 

func MustNewDigest(instanceName string, digestFunctionEnum remoteexecution.DigestFunction_Value, hash string, sizeBytes int64) Digest

MustNewDigest constructs a Digest similar to NewDigest, but never returns an error. Instead, execution will abort if the resulting instance would be degenerate. Useful for unit testing.

func NewDigestFromByteStreamReadPath 

func NewDigestFromByteStreamReadPath(path string) (Digest, remoteexecution.Compressor_Value, error)

NewDigestFromByteStreamReadPath creates a Digest from a string having one of the following formats:

- ${instanceName}/blobs/${digestFunction}/${hash}/${size} - ${instanceName}/compressed-blobs/${compressor}/${digestFunction}/${hash}/${size}

This notation is used to read files through the ByteStream service.

func NewDigestFromByteStreamWritePath 

func NewDigestFromByteStreamWritePath(path string) (Digest, remoteexecution.Compressor_Value, error)

NewDigestFromByteStreamWritePath creates a Digest from a string having one of the following formats:

- ${instanceName}/uploads/${uuid}/blobs/${digestFunction}/${hash}/${size}/${path} - ${instanceName}/uploads/${uuid}/compressed-blobs/${compressor}/${digestFunction}/${hash}/${size}/${path}

This notation is used to write files through the ByteStream service.

func (Digest) GetByteStreamReadPath 

func (d Digest) GetByteStreamReadPath(compressor remoteexecution.Compressor_Value) string

GetByteStreamReadPath converts the Digest to a string having one of the following formats:

- ${instanceName}/blobs/${digestFunction}/${hash}/${size} - ${instanceName}/compressed-blobs/${compressor}/${digestFunction}/${hash}/${size}

This notation is used to read files through the ByteStream service.

func (Digest) GetByteStreamWritePath 

func (d Digest) GetByteStreamWritePath(uuid uuid.UUID, compressor remoteexecution.Compressor_Value) string

GetByteStreamWritePath converts the Digest to a string having one of the following formats:

- ${instanceName}/uploads/${uuid}/blobs/${digestFunction}/${hash}/${size} - ${instanceName}/uploads/${uuid}/compressed-blobs/${digestFunction}/${compressor}/${hash}/${size}

This notation is used to write files through the ByteStream service.

func (Digest) GetCompactBinary 

func (d Digest) GetCompactBinary() []byte

GetCompactBinary returns a compact binary representation of the Digest, not including the instance name. The representation consists of the length of the Digest's hash, the hash in binary form, and the length of the object encoded as a variable length integer.

This representation is used by the NFSv4 server, as it needs to encode digests in file handles.

func (Digest) GetDigestFunction 

func (d Digest) GetDigestFunction() Function

GetDigestFunction returns a Function object that can be used to generate new Digest objects that use the same instance name and hashing algorithm. This method can be used in case new digests need to be derived based on an existing instance. For example, to generate a digest of an output file of a build action, given an action digest.

func (Digest) GetDigestsWithParentInstanceNames 

func (d Digest) GetDigestsWithParentInstanceNames() []Digest

GetDigestsWithParentInstanceNames returns a list of Digest objects that contain the same hash and size in bytes, but have the instance name truncated to an increasing number of components.

For example, if a digest with instance name "this/is/an/instance/name" is provided, this function will return a list of six digests, having instance names "", "this", "this/is", "this/is/an", "this/is/an/instance" and "this/is/an/instance/name".

func (Digest) GetHashBytes 

func (d Digest) GetHashBytes() []byte

GetHashBytes returns the hash of the object as a slice of bytes.

func (Digest) GetHashString 

func (d Digest) GetHashString() string

GetHashString returns the hash of the object as a string.

func (Digest) GetInstanceName 

func (d Digest) GetInstanceName() InstanceName

GetInstanceName returns the instance name of the object.

func (Digest) GetKey 

func (d Digest) GetKey(format KeyFormat) string

GetKey generates a string representation of the digest object that may be used as keys in hash tables.

func (Digest) GetProto 

func (d Digest) GetProto() *remoteexecution.Digest

GetProto encodes the digest into the format used by the remote execution protocol, so that it may be stored in messages returned to the client.

func (Digest) GetSizeBytes 

func (d Digest) GetSizeBytes() int64

GetSizeBytes returns the size of the object, in bytes.

func (Digest) NewHasher 

func (d Digest) NewHasher(expectedSizeBytes int64) hash.Hash

NewHasher creates a standard hash.Hash object that may be used to compute a checksum of data. The hash.Hash object uses the same algorithm as the one that was used to create the digest, making it possible to validate data against a digest.

The expected size can be used as a hint to create an appropriately sized hasher. If the expected size is unknown, provide math.MaxInt64.

func (Digest) String 

func (d Digest) String() string

func (Digest) ToSingletonSet 

func (d Digest) ToSingletonSet() Set

ToSingletonSet creates a Set that contains a single element that corresponds to the Digest.

func (Digest) UsesDigestFunction 

func (d Digest) UsesDigestFunction(f Function) bool

UsesDigestFunction returns true iff a Digest has the same instance name and uses the same hashing algorithm as a provided Function object.

type ExistenceCache 

type ExistenceCache struct {
	// contains filtered or unexported fields
}

ExistenceCache is a cache of digests, where entries expire once a certain duration of time has passed. It is used by ExistenceCachingBlobAccess to keep track of which objects may be omitted from FindMissing() calls.

It is safe to access ExistenceCache concurrently.

func NewExistenceCache 

func NewExistenceCache(clock clock.Clock, keyFormat KeyFormat, cacheSize int, cacheDuration time.Duration, evictionSet eviction.Set[string]) *ExistenceCache

NewExistenceCache creates a new ExistenceCache that is empty.

func NewExistenceCacheFromConfiguration 

func NewExistenceCacheFromConfiguration(configuration *pb.ExistenceCacheConfiguration, keyFormat KeyFormat, name string) (*ExistenceCache, error)

NewExistenceCacheFromConfiguration is identical to NewExistenceCache(), except that it takes a specification for the object to be created from a configuration file message.

func (*ExistenceCache) Add 

func (ec *ExistenceCache) Add(digests Set)

Add digests to the cache. These digests will automatically be removed once the duration provided to NewExistenceCache passes.

func (*ExistenceCache) RemoveExisting 

func (ec *ExistenceCache) RemoveExisting(digests Set) Set

RemoveExisting removes digests from a provided set that are present in the cache.

type Function 

type Function struct {
	// contains filtered or unexported fields
}

Function for computing new Digest objects. Function is a tuple of the REv2 instance name and hashing algorithm.

func MustNewFunction 

func MustNewFunction(instanceName string, digestFunction remoteexecution.DigestFunction_Value) Function

MustNewFunction constructs a Function similar to InstanceName.GetDigestFunction(), but never returns an error. Instead, execution will abort if the provided options are invalid. Useful for unit testing.

func (Function) GetEnumValue 

func (f Function) GetEnumValue() remoteexecution.DigestFunction_Value

GetEnumValue returns the REv2 enumeration value for the digest function.

func (Function) GetInstanceName 

func (f Function) GetInstanceName() InstanceName

GetInstanceName returns the instance name that Digest objects would use if they were created from this Function.

func (Function) NewDigest 

func (f Function) NewDigest(hash string, sizeBytes int64) (Digest, error)

NewDigest constructs a Digest object from a digest function, hash and object size. The object returned by this function is guaranteed to be non-degenerate.

func (Function) NewDigestFromProto 

func (f Function) NewDigestFromProto(digest *remoteexecution.Digest) (Digest, error)

NewDigestFromProto constructs a Digest object from a digest function and a protocol-level digest object. The object returned by this function is guaranteed to be non-degenerate.

func (Function) NewGenerator 

func (f Function) NewGenerator(expectedSizeBytes int64) *Generator

NewGenerator creates a writer that may be used to compute digests of newly created files.

The expected size can be used as a hint to create an appropriately sized hasher. If the expected size is unknown, provide math.MaxInt64.

type Generator 

type Generator struct {
	// contains filtered or unexported fields
}

Generator is a writer that may be used to compute digests of newly created files.

func (*Generator) Sum 

func (dg *Generator) Sum() Digest

Sum creates a new digest based on the data written into the Generator.

func (*Generator) Write 

func (dg *Generator) Write(p []byte) (int, error)

Write a chunk of data from a newly created file into the state of the Generator.

type InstanceName 

type InstanceName struct {
	// contains filtered or unexported fields
}

InstanceName is a simple container around REv2 instance name strings. Because instance names are embedded in URLs, the REv2 protocol places some restrictions on which instance names are valid. This type can only be instantiated for values that are valid. 

var EmptyInstanceName InstanceName

EmptyInstanceName corresponds to the instance name "". It is mainly declared to be used in places where the instance name doesn't matter (e.g., return values of functions in error cases).

func MustNewInstanceName 

func MustNewInstanceName(value string) InstanceName

MustNewInstanceName is identical to NewInstanceName, except that it panics in case the instance name is invalid. This function can be used as part of unit tests.

func NewInstanceName 

func NewInstanceName(value string) (InstanceName, error)

NewInstanceName creates a new InstanceName object that can be used to parse digests.

func NewInstanceNameFromComponents 

func NewInstanceNameFromComponents(components []string) (InstanceName, error)

NewInstanceNameFromComponents is identical to NewInstanceName, except that it takes a series of pathname components instead of a single string.

func (InstanceName) GetComponents 

func (in InstanceName) GetComponents() []string

GetComponents splits the instance name by '/' and returns each of the components. It is the inverse of NewInstanceNameFromComponents().

func (InstanceName) GetDigestFunction 

func (in InstanceName) GetDigestFunction(digestFunction remoteexecution.DigestFunction_Value, fallbackHashLength int) (Function, error)

GetDigestFunction creates a digest function object that is based on an instance name object and an REv2 digest function enumeration value.

When generating digests from within a context where a parent digest exists (e.g., on a worker that is executing an action), it is possible to call Digest.GetDigestFunction(). This function can be used when digests need to be generated outside of such contexts (e.g., on a client that is uploading actions into the Content Addressable Storage).

func (InstanceName) NewDigestFromCompactBinary 

func (in InstanceName) NewDigestFromCompactBinary(r io.ByteReader) (Digest, error)

NewDigestFromCompactBinary constructs a Digest object by reading data from a ByteReader that contains data that was generated using Digest.GetCompactBinary().

func (InstanceName) String 

func (in InstanceName) String() string

type InstanceNameMatcher 

type InstanceNameMatcher func(i InstanceName) bool

InstanceNameMatcher is a function callback type that corresponds with the signature of InstanceNameTrie.ContainsPrefix. It can be used in places where an InstanceNameTrie is used as a simple set.

type InstanceNamePatcher 

type InstanceNamePatcher interface {
	PatchInstanceName(i InstanceName) InstanceName

	PatchDigest(d Digest) Digest
	UnpatchDigest(d Digest) Digest
}

InstanceNamePatcher can be used to insert, strip or change a prefix of an instance name in InstanceName and Digest objects. 

var NoopInstanceNamePatcher InstanceNamePatcher = noopInstanceNamePatcher{}

NoopInstanceNamePatcher is an InstanceNamePatcher that performs no substitutions on the instance name.

func NewInstanceNamePatcher 

func NewInstanceNamePatcher(oldPrefix, newPrefix InstanceName) InstanceNamePatcher

NewInstanceNamePatcher creates an InstanceNamePatcher that replaces a given prefix of an instance name with another value. It is not valid to apply the resulting InstanceNamePatcher against objects that don't have an instance name starting with the provided prefix.

type InstanceNameTrie 

type InstanceNameTrie struct {
	// contains filtered or unexported fields
}

InstanceNameTrie implements a trie (prefix tree) for instance names. It can be used to demultiplex remote execution requests based on the instance name.

For every key stored in the trie, an integer value is tracked. This can, for example, be used by the caller to look up a corresponding value in a contiguous list.

func NewInstanceNameTrie 

func NewInstanceNameTrie() *InstanceNameTrie

NewInstanceNameTrie creates a new InstanceNameTrie that is initialized with no elements.

func (*InstanceNameTrie) ContainsExact 

func (it *InstanceNameTrie) ContainsExact(i InstanceName) bool

ContainsExact returns whether the trie contains an instance name that is exactly the same as the one provided.

func (*InstanceNameTrie) ContainsPrefix 

func (it *InstanceNameTrie) ContainsPrefix(i InstanceName) bool

ContainsPrefix returns whether the trie contains one or more instance names that are a prefix of the one provided.

func (*InstanceNameTrie) GetExact 

func (it *InstanceNameTrie) GetExact(i InstanceName) int

GetExact returns the value associated with the instance name. If none of the instance names provided to Set() are exactly the same as the instance name provided to GetExact(), this function returns -1.

func (*InstanceNameTrie) GetLongestPrefix 

func (it *InstanceNameTrie) GetLongestPrefix(i InstanceName) int

GetLongestPrefix returns the value associated with the longest matching instance name prefix. If none of the instance names provided to Set() are a prefix of the instance name provided to GetLongestPrefix(), this function returns -1.

func (*InstanceNameTrie) Remove 

func (it *InstanceNameTrie) Remove(i InstanceName) bool

Remove a value associated with an instance name. This function returns whether removing the instance name has caused the trie to become empty.

func (*InstanceNameTrie) Set 

func (it *InstanceNameTrie) Set(i InstanceName, value int)

Set an instance name in the trie to a given integer value.

type KeyFormat 

type KeyFormat int

KeyFormat is an enumeration type that determines the format of object keys returned by Digest.GetKey(). 

const (
	// KeyWithoutInstance lets Digest.GetKey() return a key that
	// does not include the name of the instance; only the hash and
	// the size.
	KeyWithoutInstance KeyFormat = iota
	// KeyWithInstance lets Digest.GetKey() return a key that
	// includes the hash, size and instance name.
	KeyWithInstance
)

func (KeyFormat) Combine 

func (kf KeyFormat) Combine(other KeyFormat) KeyFormat

Combine two KeyFormats into one, picking the format that contains the most information.

This function is used extensively by NewBlobAccessFromConfiguration() to ensure that the right KeyFormat is picked based on the behavior of two or more backing BlobAccess instances.

type Set 

type Set struct {
	// contains filtered or unexported fields
}

Set of digests. Sets are immutable and can be created using SetBuilder.

func GetDifferenceAndIntersection 

func GetDifferenceAndIntersection(setA, setB Set) (onlyA, both, onlyB Set)

GetDifferenceAndIntersection partitions the elements stored in sets A and B across three resulting sets: one containing the elements present only in A, one containing the elements present in both A and B, and one containing thelements present only in B.

func GetUnion 

func GetUnion(sets []Set) Set

GetUnion merges all of the elements stored in a list of sets into a single resulting set. This implementation uses a k-way merging algorithm.

func (Set) Empty 

func (s Set) Empty() bool

Empty returns true if the set contains zero elements.

func (Set) First 

func (s Set) First() (Digest, bool)

First returns the first element stored in the set. The boolean return value denotes whether the operation was successful (i.e., the set is non-empty).

func (Set) Items 

func (s Set) Items() []Digest

Items returns a sorted list of all elements stored within the set.

func (Set) Length 

func (s Set) Length() int

Length returns the number of elements stored in the set.

func (Set) PartitionByInstanceName 

func (s Set) PartitionByInstanceName() []Set

PartitionByInstanceName partitions the elements stored in a set into a list of sets, each containing elements of a single REv2 instance name.

All resulting sets are guaranteed to be non-empty. The order in which they are returned corresponds with the order in which their first element was stored in the original set.

func (Set) RemoveEmptyBlob 

func (s Set) RemoveEmptyBlob() Set

RemoveEmptyBlob returns a copy of the set that has all of the entries corresponding with the empty blob removed.

type SetBuilder 

type SetBuilder struct {
	// contains filtered or unexported fields
}

SetBuilder is a builder for Set objects.

func NewSetBuilder 

func NewSetBuilder() SetBuilder

NewSetBuilder creates a SetBuilder that contains no initial elements.

func (SetBuilder) Add 

func (sb SetBuilder) Add(digest Digest) SetBuilder

Add a single element to the Set that is being built by the SetBuilder.

func (SetBuilder) Build 

func (sb SetBuilder) Build() Set

Build the Set containing the Digests provided to Add().

func (SetBuilder) Length 

func (sb SetBuilder) Length() int

Length returns the number of elements that the Set would contain if built.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.