cache

package
v0.0.0-...-fc78dae Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 6, 2024 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package cache implements local cached storage of part data.

The cache is accessed using an interface defined as MoteCache, and implemented with BottleFileCache Generally, the cache is intended to be used in a pull-through fashion, with data being transferred first to the cache and then automatically to its final destination. Items in the cache are identified using a digest, though the source of those digests are generally within an oci descriptor.

Index

Constants

View Source 
const CommitCopy = 1

CommitCopy copies a file, leaving the original in place.

View Source 
const CommitMove = 2

CommitMove Moves a file, removing the original.

Variables

View Source 
var (
	// ErrNotFound is an error returned when attempting to access a mote that isn't present in the cache.
	ErrNotFound = errors.New("specified item is not found in the cache")

	// ErrSizeMismatch is an error raised when a file write is terminated before expected.
	ErrSizeMismatch = errors.New("size of transferred item does not match expected value")
)

Functions

func LocateLayer 

func LocateLayer(ctx context.Context, bic BIC, desc ocispec.Descriptor, destRef ref.Ref, matchReg bool) []ref.Ref

LocateLayer returns a set of bottle Refs containing a list of known locations for the provided layer descriptor. The layer descriptor should be manifest-level (eg. possibly compressed/encrypted). Candidates returned will be located on the destination registry if matchReg is true.

func LocateLayerDigest 

func LocateLayerDigest(ctx context.Context, bic BIC, dgst digest.Digest, destRef ref.Ref, matchReg bool) []ref.Ref

LocateLayerDigest performs a layer location query based on a digest representation of the layer digest. This works the same as LocateLayer.

func RecordLayerSource 

func RecordLayerSource(ctx context.Context, bic BIC, desc ocispec.Descriptor, srcRef ref.Ref)

RecordLayerSource adds the layer described by desc to the local blob info cache, with the provided srcRef as the known source location. The desc should describe the manifest-level layer (eg, possibly compressed/encrypted).

func TransportFromImageName 

func TransportFromImageName(refString string) string

TransportFromImageName extracts a transport string from the provided ref string, if one exists. This allows filtering on location, such as images located in an oci-layout dir, archive, etc. If a transport string isn't found in the refString, (format is transport:ref), or if an unknown transport is specified, "oci" is returned.

Types

type BIC 

type BIC bicbackend.BlobInfoCache

BIC is a type alias for a blob info cache.

func NewCache 

func NewCache(cachePath string) BIC

NewCache returns the active blob info cache. Opening and initialization is delayed until access. If cachePath is an empty string, a memory cache is returned, otherwise it is a persistent cache backed by boltdb.

type BottleFileCache 

type BottleFileCache struct {
	DigestAlgorithm digest.Algorithm

	LinkOnReadDisabled bool      // Whether or not to create a hard link during read operations to prevent locking files
	FallbackCache      MoteCache // A secondary cache to check if the primary cache does not contain a requested file.  This fallback is considered only on read operations. Caution, avoid graph cycles!
	// contains filtered or unexported fields
}

BottleFileCache is an implementation of MoteCache for dealing with data bottle caching of blobs. No metadata is maintained for the motes, relying on file existence to identify items present in the cache.

func NewBottleFileCache 

func NewBottleFileCache(path string, opts ...BottleFileCacheOpt) *BottleFileCache

NewBottleFileCache creates a new bottle cache object for managing data bottle cache. This does not load or create the cache on disk, use Initialize to do so.

func (*BottleFileCache) CommitMote 

func (bc *BottleFileCache) CommitMote(srcpath string, mote Mote, commitMode int) error

CommitMote adds or finalizes an existing file into the cache. Use CommitCopy mode to keep the original file on commit Use CommitMove mode to remove the original file (using Rename) WARNING: this function is re-entered on CommitCopy, because MoteWriter is used to synchronize writes to the cache, which in turn calls this function with CommitMove mode to finalize the update.

func (*BottleFileCache) CreateMote 

func (bc *BottleFileCache) CreateMote(dgst digest.Digest, mediaType string, size int64) (Mote, error)

CreateMote returns a new pending Mote based on the BottleFileCache settings.

func (*BottleFileCache) Find 

func (bc *BottleFileCache) Find(dgst digest.Digest) (MoteInfo, bool)

Find returns the mote information interface for a known mote. since BottleFileCache does not retain metadata information for motes, this will only contain digest and file size for the mote.

func (*BottleFileCache) Initialize 

func (bc *BottleFileCache) Initialize() error

Initialize checks if a cache exists at the cache path path, and creates one if necessary.

func (*BottleFileCache) IsDirty 

func (bc *BottleFileCache) IsDirty() bool

IsDirty always returns false for BottleFileCache -- no way to detect if there have been changes.

func (*BottleFileCache) MoteExists 

func (bc *BottleFileCache) MoteExists(dgst digest.Digest) bool

MoteExists returns true if the given item appears in the cache, This does not check the pending motes.

func (*BottleFileCache) MoteReader 

func (bc *BottleFileCache) MoteReader(dgst digest.Digest) (io.ReadCloser, error)

MoteReader returns a MoteReader for reading from a cache blob. Before opening the blob, the file is hard linked to a temporary file name, and the temporary file is removed upon close.

func (*BottleFileCache) MoteReaderAt 

func (bc *BottleFileCache) MoteReaderAt(dgst digest.Digest) (ReaderAt, error)

MoteReaderAt returns an io.ReaderAt for offset based reading.

func (*BottleFileCache) MoteRef 

func (bc *BottleFileCache) MoteRef(dgst digest.Digest) string

MoteRef for bottlefilecache returns the expected file path for a mote digest. The file is not checked for existence.

func (*BottleFileCache) MoteWriter 

func (bc *BottleFileCache) MoteWriter(dgst digest.Digest) (io.WriteCloser, error)

MoteWriter returns a MoteWriter for writing to a cache blob. The write is done to a temporary location, and an enclosed Close handler is used to finalize the cache addition.

func (*BottleFileCache) Prune 

func (bc *BottleFileCache) Prune(ctx context.Context, maxSize int64) error

Prune removes bottle items until the total size of the cache is less than or equal to maxSize.

func (*BottleFileCache) Refresh 

func (bc *BottleFileCache) Refresh() (bool, error)

Refresh is a null action for BottleFileCache, false is returned to indicate that no update process needs to be performed.

func (*BottleFileCache) RemoveMote 

func (bc *BottleFileCache) RemoveMote(dgst digest.Digest) error

RemoveMote removes a mote from the collection of cached motes.

func (*BottleFileCache) Update 

func (bc *BottleFileCache) Update() error

Update does nothing for BottleFileCache, as there is no state to update.

type BottleFileCacheOpt 

type BottleFileCacheOpt func(*BottleFileCache)

BottleFileCacheOpt is a functional option type for configuring a bottle cache on creation.

func DisableLinkOnRead 

func DisableLinkOnRead() BottleFileCacheOpt

DisableLinkOnRead causes the BottleFileCache to not create hard links when creating a mote reader.

func WithFallbackCache 

func WithFallbackCache(cache MoteCache) BottleFileCacheOpt

WithFallbackCache enables a fallback MoteCache that can provide a secondary source for motes not currently in the cache, such as an HTTP source for retrieval. This only used for a read source cache, and additionally does not add items to the primary cache location when the fallback cache is used as the stand in reader.

func WithKeyAlgorithm 

func WithKeyAlgorithm(alg digest.Algorithm) BottleFileCacheOpt

WithKeyAlgorithm allows an override to create a cache using a different hashing algorithm than the default. The algorithm is one of the open container go-digest algorithm constants (strings).

type BytesTracker 

type BytesTracker struct {

	// Total number of bytes of bytes seen so far
	Total int64

	// Total number of bytes seen so far with duplicates removed
	Deduplicated int64
	// contains filtered or unexported fields
}

BytesTracker is an object for tracking digests seen, total bytes seen, and deduplication size.

func (*BytesTracker) Add 

func (bt *BytesTracker) Add(desc ocispec.Descriptor)

Add adds a digest to the BytesTracker map and computes the total size and deduplicated size.

type LayerBlobInfo 

type LayerBlobInfo struct {
	// contains filtered or unexported fields
}

LayerBlobInfo is a structure implementing the SourceProvider interface, enabling lookup and server-to server transfer for layers based on recorded source information in blobinfocache.

func (*LayerBlobInfo) GetSources 

func (lbi *LayerBlobInfo) GetSources(ctx context.Context, bic BIC) map[digest.Digest][]string

GetSources returns a map of layerID to known source list for all layerIDs in a LayerBlobInfo, implementing the SourceProvider interface.

type Mote 

type Mote struct {
	// Digest is a digest identifier for an item
	Digest digest.Digest
	// MediaType is the oci media type
	MediaType string
	// DataSize is the real size in bytes of an item
	DataSize int64
	// contains filtered or unexported fields
}

Mote is a representation of a single item in a cache, an abstraction that may be backed by a file, memory, or other data blob. Access to motes and manipulation of them should be done via the Mote* interfaces.

func (Mote) Close 

func (m Mote) Close() error

Close implements the io.Closer interface, optionally forwaring to a handler for finalizing a read/write operation. For writers, the close handler should perform atomic or locked cache updates.

func (Mote) GetDigest 

func (m Mote) GetDigest() digest.Digest

GetDigest returns the digest of the mote.

func (Mote) GetMediaType 

func (m Mote) GetMediaType() string

GetMediaType returns the known media type for the mote.

func (Mote) Read 

func (m Mote) Read(p []byte) (n int, err error)

Read implements the io.Reader interface, optionally forwarding to a handler for reading data from a cache mote.

func (Mote) ReadAt 

func (m Mote) ReadAt(p []byte, off int64) (n int, err error)

ReadAt implements the io.ReaderAt interface, opionally forwarding to a handler for performing the functionality.

func (Mote) Size 

func (m Mote) Size() int64

Size returns the size of the mote.

func (Mote) Write 

func (m Mote) Write(p []byte) (n int, err error)

Write implements the io.Writer interface, optionally forwarding to a handler for writing data to a cache mote. Note, a writer should write to a temporary location, then perform a cache update (e.g. movefile) during the close operation.

type MoteCache 

type MoteCache interface {
	// Initialize should load metadata for a cache, or instantiate a new cache if one does not exist.  The initialized
	// or created cache is returned (which may not be the one that was used to call initialize
	Initialize() error

	// IsDirty returns true if the cache has been updated since the last Initialization
	IsDirty() bool

	// Refresh triggers a reload of cache metadata from the index.  This should be called prior to performing an update.
	// The refresh operation should return true if data was updated, and false if not. An error should be returned if
	// the process fails
	Refresh() (bool, error)

	// MoteExists should return true if an item matching digest exists in the cache
	MoteExists(dgst digest.Digest) bool

	// MoteWriter returns an io.WriteCloser interface for writing data to an item. It is expected that the digest is
	// known to the cache at the time of requesting a writer (CreateMote called first), so this returns ErrNotFound if
	// the mote is not found.
	MoteWriter(dgst digest.Digest) (io.WriteCloser, error)

	// MoteReader returns an io.ReadCloser interface for reading data from a cached item based on its digest
	MoteReader(dgst digest.Digest) (io.ReadCloser, error)

	// MoteReaderAt returns an io.ReadAtCloser interface for reading data from a cached item based on its digest
	MoteReaderAt(dgst digest.Digest) (ReaderAt, error)

	// Find returns a mote information interface based on a digest search, and true if the item is found, false if not
	Find(dgst digest.Digest) (MoteInfo, bool)

	// MoteRef returns information about a mote location or reference, for instance an explicit file path
	MoteRef(dgst digest.Digest) string

	// CommitMote finalizes the addition of a mote or update to the cache. ref can be a source file path or other
	// reference information, while mote contains the mote information to be added to the index.  This silently does
	// nothing if no update is required.
	CommitMote(ref string, mote Mote, commitMode int) error

	// CreateMote creates a new item and returns an error if not successful
	CreateMote(dgst digest.Digest, mediaType string, size int64) (Mote, error)

	// RemoveMote removes an item and returns an error if not successful
	RemoveMote(dgst digest.Digest) error

	// Prune removes cached items until the size is less than or equal to maxSize
	Prune(ctx context.Context, maxSize int64) error
}

MoteCache is an interface for working with cache storage.

type MoteInfo 

type MoteInfo interface {
	// GetDigest retrieves the digest of a cached item.  This is not a calculated value, but instead refers to the
	// digest identifier of the cached item (Though the two should match)
	GetDigest() digest.Digest
	// Size retrieves the size of the data in bytes for the cached item
	Size() int64
	// GetMediaType returns a string identifying the media type for the cached item
	GetMediaType() string
}

MoteInfo is an interface representing the public information available for a mote.

type NilCache 

type NilCache struct {
}

NilCache is an empty implementation of MoteCache that provides empty functionality for cases when caching is disabled.

func (*NilCache) CommitMote 

func (nc *NilCache) CommitMote(ref string, mote Mote, commitMode int) error

CommitMote does nothing.

func (*NilCache) CreateMote 

func (nc *NilCache) CreateMote(dgst digest.Digest, mediaType string, size int64) (Mote, error)

CreateMote creates a mote with the provided info. The mote created with this contains the provided information, but is otherwise non functional.

func (*NilCache) Find 

func (nc *NilCache) Find(dgst digest.Digest) (MoteInfo, bool)

Find returns no info, and false to indicate nothing was found.

func (*NilCache) Initialize 

func (nc *NilCache) Initialize() error

Initialize does nothing.

func (*NilCache) IsDirty 

func (nc *NilCache) IsDirty() bool

IsDirty returns false.

func (*NilCache) MoteExists 

func (nc *NilCache) MoteExists(dgst digest.Digest) bool

MoteExists returns false.

func (*NilCache) MoteReader 

func (nc *NilCache) MoteReader(dgst digest.Digest) (io.ReadCloser, error)

MoteReader returns an empty mote.

func (*NilCache) MoteReaderAt 

func (nc *NilCache) MoteReaderAt(dgst digest.Digest) (ReaderAt, error)

MoteReaderAt returns an empty mote.

func (*NilCache) MoteRef 

func (nc *NilCache) MoteRef(dgst digest.Digest) string

MoteRef returns an empty string.

func (*NilCache) MoteWriter 

func (nc *NilCache) MoteWriter(dgst digest.Digest) (io.WriteCloser, error)

MoteWriter returns an empty mote.

func (*NilCache) Prune 

func (nc *NilCache) Prune(ctx context.Context, maxSize int64) error

Prune does nothing.

func (*NilCache) Refresh 

func (nc *NilCache) Refresh() (bool, error)

Refresh does nothing.

func (*NilCache) RemoveMote 

func (nc *NilCache) RemoveMote(dgst digest.Digest) error

RemoveMote does nothing.

type ReaderAt 

type ReaderAt interface {
	io.ReaderAt
	io.Closer
	Size() int64
}

ReaderAt extends the standard io.ReaderAt interface with reporting of Size and io.Closer.

type SourceProvider 

type SourceProvider interface {
	// GetSources returns a map of digests to a list of OCI references (in string format) where each digest is known to
	// exist. This allows the content to be retrieved from a remote source.
	GetSources() map[digest.Digest][]string
}

SourceProvider is an interface enabling the retrieval of file sources, which is a mapping of digest string to list of OCI Refs in string format, allowing content to be retrieved from a remote source.

Source Files

View all Source files

Directories

Path Synopsis
Package bicbackend implements the management and storage of blob info cache data, including boltdb and memory implementations.
 Package bicbackend implements the management and storage of blob info cache data, including boltdb and memory implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.