internal

Documentation

Index

• Variables
• func MarshalWithSHA256(pm proto.Message) ([]byte, error)
• func UnmarshalWithSHA256(buf []byte, pm proto.Message) error
• type Fetcher
• type InstanceCache
  • func (c *InstanceCache) Close(ctx context.Context)
  • func (c *InstanceCache) GC(ctx context.Context)
  • func (c *InstanceCache) HasPendingFetches() bool
  • func (c *InstanceCache) Launch(ctx context.Context)
  • func (c *InstanceCache) RequestInstances(ctx context.Context, reqs []*InstanceRequest)
  • func (c *InstanceCache) WaitInstance() (res *InstanceResult)
• type InstanceRequest
• type InstanceResult
• type TagCache
  • func NewTagCache(fs fs.FileSystem, service string) *TagCache
  • func (c *TagCache) AddExtractedObjectRef(ctx context.Context, pin common.Pin, fileName string, ref *api.ObjectRef) error
  • func (c *TagCache) AddTag(ctx context.Context, pin common.Pin, tag string) error
  • func (c *TagCache) ResolveExtractedObjectRef(ctx context.Context, pin common.Pin, fileName string) (*api.ObjectRef, error)
  • func (c *TagCache) ResolveTag(ctx context.Context, pkg, tag string) (pin common.Pin, err error)
  • func (c *TagCache) Save(ctx context.Context) error

Variables

var ErrUnknownSHA256 = errors.New("no sha256 is recorded in the file")

ErrUnknownSHA256 indicates the deserialized message doesn't have SHA256 set.

This can happen when deserializing records in the old format.

Functions

func MarshalWithSHA256

func MarshalWithSHA256(pm proto.Message) ([]byte, error)

MarshalWithSHA256 serializes proto message to bytes, calculates SHA256 checksum of it, and returns serialized envelope that contains both.

UnmarshalWithSHA256 can then be used to verify SHA256 and deserialized the original object.

func UnmarshalWithSHA256

func UnmarshalWithSHA256(buf []byte, pm proto.Message) error

UnmarshalWithSHA256 is reverse of MarshalWithSHA256.

It checks SHA256 checksum and deserializes the object if it matches the blob.

If the expected SHA256 is not available in 'buf', returns ErrUnknownSHA256. This can happen when reading blobs in old format that used SHA1.

Types

type Fetcher

type Fetcher func(ctx context.Context, pin common.Pin, f io.WriteSeeker) error

Fetcher knows how to populate a file in the cache given its pin.

It must check the hash of the downloaded package matches the pin.

type InstanceCache

type InstanceCache struct {
	// FS is a root of the cache directory.
	FS fs.FileSystem

	// Tmp, if true, indicates this is a temporary instance cache.
	//
	// It is a cache of downloaded, but not yet installed packages. It has a
	// property that cached files self-destruct after being closed and the entire
	// cache directory is removed in Close().
	Tmp bool

	// Fetcher is used to fetch files into the cache on cache misses.
	//
	// It must check the hash of the downloaded package matches the pin.
	Fetcher Fetcher

	// ParallelDownloads limits how many parallel fetches can happen at once.
	//
	// The zero value means to do fetches in a blocking way in WaitInstance.
	ParallelDownloads int
	// contains filtered or unexported fields
}

InstanceCache is a file-system-based, thread-safe, LRU cache of instances.

It also knows how to populate the cache using the given Fetcher callback, fetching multiple instances concurrently.

Does not validate instance hashes; it is the responsibility of the supplied Fetcher callback.

func (*InstanceCache) Close

func (c *InstanceCache) Close(ctx context.Context)

Close shuts down goroutines started in Launch and cleans up temp files.

The caller should ensure there's no pending fetches before making this call.

func (*InstanceCache) GC

func (c *InstanceCache) GC(ctx context.Context)

GC opportunistically purges entries that haven't been touched for too long.

func (*InstanceCache) HasPendingFetches

func (c *InstanceCache) HasPendingFetches() bool

HasPendingFetches is true if there's some uncompleted InstanceRequest whose completion notification will (eventually) unblock WaitInstance.

func (*InstanceCache) Launch

func (c *InstanceCache) Launch(ctx context.Context)

Launch prepares the cache for fetching packages in background.

Must be called before RequestInstances.

func (*InstanceCache) RequestInstances

func (c *InstanceCache) RequestInstances(ctx context.Context, reqs []*InstanceRequest)

RequestInstances enqueues requests to fetch instances into the cache.

The results can be waited upon using WaitInstance() method. Each enqueued InstanceRequest will get an exactly one InstanceResponse (perhaps with an error inside). The order of responses may not match the order of requests.

func (*InstanceCache) WaitInstance

func (c *InstanceCache) WaitInstance() (res *InstanceResult)

WaitInstance blocks until some InstanceRequest is available.

type InstanceRequest

type InstanceRequest struct {
	Context context.Context    // carries the cancellation signal
	Done    context.CancelFunc // called right before the result is enqueued
	Pin     common.Pin         // identifies the instance to fetch
	Open    bool               // true to return it as a pkg.Instance as opposed to pkg.Source
	State   any                // passed to the InstanceResult as is
}

InstanceRequest is passed to RequestInstances.

type InstanceResult

type InstanceResult struct {
	Context  context.Context // copied from the InstanceRequest
	Err      error           // non-nil if failed to obtain the instance
	Source   pkg.Source      // set only if Open was false, must be closed by the caller
	Instance pkg.Instance    // set only if Open was true, must be closed by the caller
	State    any             // copied from the InstanceRequest
}

InstanceResult is a result of a completed InstanceRequest.

type TagCache

type TagCache struct {
	// contains filtered or unexported fields
}

TagCache provides a mapping (package name, tag) -> instance ID.

This mapping is safe to cache because tags are not detachable: once a tag is successfully resolved to an instance ID it is guaranteed to resolve to same instance ID later or not resolve at all (e.g. if one tag is attached to multiple instances, in which case the tag is misused anyway). In any case, returning a cached instance ID does make sense. The primary purpose of this cache is to avoid round trips to the service to increase reliability of 'cipd ensure' calls that use only tags to specify versions. It happens to be the most common case of 'cipd ensure' usage by far.

Additionally, this TagCache stores a mapping of (pin, file_name) -> encode(ObjectRef of extracted file) to assist in the `selfupdate` flow.

Whenever selfupdate resolves what CIPD package instance ID (pin) it SHOULD be at, it looks at '(pin, 'cipd') => binary hash' map to figure out what hash the client itself SHOULD have for this instance ID. The client then calculates the hash of itself to see if it's actually already at that instance ID.

func NewTagCache

func NewTagCache(fs fs.FileSystem, service string) *TagCache

NewTagCache initializes TagCache.

fs will be the root of the cache. It will be searched for tagcache.db file.

func (*TagCache) AddExtractedObjectRef

func (c *TagCache) AddExtractedObjectRef(ctx context.Context, pin common.Pin, fileName string, ref *api.ObjectRef) error

AddExtractedObjectRef records that fileName extracted from the package at the given pin has the given hash.

The hash is represented as ObjectRef, which is a tuple (hash algo, hex digest).

Call 'Save' later to persist these changes to the cache file on disk.

func (*TagCache) AddTag

func (c *TagCache) AddTag(ctx context.Context, pin common.Pin, tag string) error

AddTag records that (pin.PackageName, tag) maps to pin.InstanceID.

Call 'Save' later to persist these changes to the cache file on disk.

func (*TagCache) ResolveExtractedObjectRef

func (c *TagCache) ResolveExtractedObjectRef(ctx context.Context, pin common.Pin, fileName string) (*api.ObjectRef, error)

ResolveExtractedObjectRef returns ObjectRef or nil if that file is not in the cache.

Returns error if the cache can't be read.

func (*TagCache) ResolveTag

func (c *TagCache) ResolveTag(ctx context.Context, pkg, tag string) (pin common.Pin, err error)

ResolveTag returns cached tag or empty Pin{} if such tag is not in the cache.

Returns error if the cache can't be read.

func (*TagCache) Save

func (c *TagCache) Save(ctx context.Context) error

Save stores all pending cache updates to the file system.

It effectively resets the object to the initial state.