cache

package
v0.4.210-rc.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 30, 2023 License: BSD-3-Clause Imports: 29 Imported by: 0

Documentation

Overview

Package cache implements a build artifact cache.

Index

Constants

View Source 
const HashSize = 32

HashSize is the number of bytes in a hash.

Variables

View Source 
var DebugTest = false

DebugTest is set when GODEBUG=gocachetest=1 is in the environment.

Functions

func DefaultDir 

func DefaultDir() string

DefaultDir returns the effective GOCACHE setting. It returns "off" if the cache is disabled.

func FileHash 

func FileHash(file string) ([HashSize]byte, error)

FileHash returns the hash of the named file. It caches repeated lookups for a given file, and the cache entry for a file can be initialized using SetFileHash. The hash used by FileHash is not the same as the hash used by NewHash.

func PutBytes added in v0.7.2202 

func PutBytes(c Cache, id ActionID, data []byte) error

PutBytes stores the given bytes in the cache as the output for the action ID.

func SetFileHash 

func SetFileHash(file string, sum [HashSize]byte)

SetFileHash sets the hash returned by FileHash for file.

Types

type ActionID 

type ActionID [HashSize]byte

An ActionID is a cache action key, the hash of a complete description of a repeatable computation (command line, environment variables, input file contents, executable contents).

func Subkey 

func Subkey(parent ActionID, desc string) ActionID

Subkey returns an action ID corresponding to mixing a parent action ID with a string description of the subkey.

type Cache 

type Cache interface {
	// Get returns the cache entry for the provided ActionID.
	// On miss, the error type should be of type *entryNotFoundError.
	//
	// After a success call to Get, OutputFile(Entry.OutputID) must
	// exist on disk for until Close is called (at the end of the process).
	Get(ActionID) (Entry, error)

	// Put adds an item to the cache.
	//
	// The seeker is only used to seek to the beginning. After a call to Put,
	// the seek position is not guaranteed to be in any particular state.
	//
	// As a special case, if the ReadSeeker is of type noVerifyReadSeeker,
	// the verification from GODEBUG=goverifycache=1 is skipped.
	//
	// After a success call to Get, OutputFile(Entry.OutputID) must
	// exist on disk for until Close is called (at the end of the process).
	Put(ActionID, io.ReadSeeker) (_ OutputID, size int64, _ error)

	// Close is called at the end of the go process. Implementations can do
	// cache cleanup work at this phase, or wait for and report any errors from
	// background cleanup work started earlier. Any cache trimming should in one
	// process should not violate cause the invariants of this interface to be
	// violated in another process. Namely, a cache trim from one process should
	// not delete an ObjectID from disk that was recently Get or Put from
	// another process. As a rule of thumb, don't trim things used in the last
	// day.
	Close() error

	// OutputFile returns the path on disk where OutputID is stored.
	//
	// It's only called after a successful get or put call so it doesn't need
	// to return an error; it's assumed that if the previous get or put succeeded,
	// it's already on disk.
	OutputFile(OutputID) string

	// FuzzDir returns where fuzz files are stored.
	FuzzDir() string
}

Cache is the interface as used by the cmd/go.

func Default 

func Default() Cache

Default returns the default cache to use. It never returns nil.

type DiskCache added in v0.7.2202 

type DiskCache struct {
	// contains filtered or unexported fields
}

A Cache is a package cache, backed by a file system directory tree.

func Open 

func Open(dir string) (*DiskCache, error)

Open opens and returns the cache in the given directory.

It is safe for multiple processes on a single machine to use the same cache directory in a local file system simultaneously. They will coordinate using operating system file locks and may duplicate effort but will not corrupt the cache.

However, it is NOT safe for multiple processes on different machines to share a cache directory (for example, if the directory were stored in a network file system). File locking is notoriously unreliable in network file systems and may not suffice to protect the cache.

func (*DiskCache) Close added in v0.7.2202 

func (c *DiskCache) Close() error

func (*DiskCache) FuzzDir added in v0.7.2202 

func (c *DiskCache) FuzzDir() string

FuzzDir returns a subdirectory within the cache for storing fuzzing data. The subdirectory may not exist.

This directory is managed by the internal/fuzz package. Files in this directory aren't removed by the 'go clean -cache' command or by Trim. They may be removed with 'go clean -fuzzcache'.

TODO(#48526): make Trim remove unused files from this directory.

func (*DiskCache) Get added in v0.7.2202 

func (c *DiskCache) Get(id ActionID) (Entry, error)

Get looks up the action ID in the cache, returning the corresponding output ID and file size, if any. Note that finding an output ID does not guarantee that the saved file for that output ID is still available.

func (*DiskCache) OutputFile added in v0.7.2202 

func (c *DiskCache) OutputFile(out OutputID) string

OutputFile returns the name of the cache file storing output with the given OutputID.

func (*DiskCache) Put added in v0.7.2202 

func (c *DiskCache) Put(id ActionID, file io.ReadSeeker) (OutputID, int64, error)

Put stores the given output in the cache as the output for the action ID. It may read file twice. The content of file must not change between the two passes.

func (*DiskCache) Trim added in v0.7.2202 

func (c *DiskCache) Trim() error

Trim removes old cache entries that are likely not to be reused.

type Entry 

type Entry struct {
	OutputID OutputID
	Size     int64
	Time     time.Time // when added to cache
}

func GetBytes added in v0.7.2202 

func GetBytes(c Cache, id ActionID) ([]byte, Entry, error)

GetBytes looks up the action ID in the cache and returns the corresponding output bytes. GetBytes should only be used for data that can be expected to fit in memory.

func GetFile added in v0.7.2202 

func GetFile(c Cache, id ActionID) (file string, entry Entry, err error)

GetFile looks up the action ID in the cache and returns the name of the corresponding data file.

func GetMmap added in v0.7.2202 

func GetMmap(c Cache, id ActionID) ([]byte, Entry, error)

GetMmap looks up the action ID in the cache and returns the corresponding output bytes. GetMmap should only be used for data that can be expected to fit in memory.

type Hash 

type Hash struct {
	// contains filtered or unexported fields
}

A Hash provides access to the canonical hash function used to index the cache. The current implementation uses salted SHA256, but clients must not assume this.

func NewHash 

func NewHash(name string) *Hash

NewHash returns a new Hash. The caller is expected to Write data to it and then call Sum.

func (*Hash) Sum 

func (h *Hash) Sum() [HashSize]byte

Sum returns the hash of the data written previously.

func (*Hash) Write 

func (h *Hash) Write(b []byte) (int, error)

Write writes data to the running hash.

type OutputID 

type OutputID [HashSize]byte

An OutputID is a cache output key, the hash of an output of a computation.

func PutNoVerify added in v0.7.2202 

func PutNoVerify(c Cache, id ActionID, file io.ReadSeeker) (OutputID, int64, error)

PutNoVerify is like Put but disables the verify check when GODEBUG=goverifycache=1 is set. It is meant for data that is OK to cache but that we expect to vary slightly from run to run, like test output containing times and the like.

type ProgCache added in v0.7.2202 

type ProgCache struct {
	// contains filtered or unexported fields
}

ProgCache implements Cache via JSON messages over stdin/stdout to a child helper process which can then implement whatever caching policy/mechanism it wants.

See https://github.com/golang/go/issues/59719

func (*ProgCache) Close added in v0.7.2202 

func (c *ProgCache) Close() error

func (*ProgCache) FuzzDir added in v0.7.2202 

func (c *ProgCache) FuzzDir() string

func (*ProgCache) Get added in v0.7.2202 

func (c *ProgCache) Get(a ActionID) (Entry, error)

func (*ProgCache) OutputFile added in v0.7.2202 

func (c *ProgCache) OutputFile(o OutputID) string

func (*ProgCache) Put added in v0.7.2202 

func (c *ProgCache) Put(a ActionID, file io.ReadSeeker) (_ OutputID, size int64, _ error)

type ProgCmd added in v0.7.2202 

type ProgCmd string

ProgCmd is a command that can be issued to a child process.

If the interface needs to grow, we can add new commands or new versioned commands like "get2".

type ProgRequest added in v0.7.2202 

type ProgRequest struct {
	// ID is a unique number per process across all requests.
	// It must be echoed in the ProgResponse from the child.
	ID int64

	// Command is the type of request.
	// The cmd/go tool will only send commands that were declared
	// as supported by the child.
	Command ProgCmd

	// ActionID is non-nil for get and puts.
	ActionID []byte `json:",omitempty"` // or nil if not used

	// ObjectID is set for Type "put" and "output-file".
	ObjectID []byte `json:",omitempty"` // or nil if not used

	// Body is the body for "put" requests. It's sent after the JSON object
	// as a base64-encoded JSON string when BodySize is non-zero.
	// It's sent as a separate JSON value instead of being a struct field
	// send in this JSON object so large values can be streamed in both directions.
	// The base64 string body of a ProgRequest will always be written
	// immediately after the JSON object and a newline.
	Body io.Reader `json:"-"`

	// BodySize is the number of bytes of Body. If zero, the body isn't written.
	BodySize int64 `json:",omitempty"`
}

ProgRequest is the JSON-encoded message that's sent from cmd/go to the GOCACHEPROG child process over stdin. Each JSON object is on its own line. A ProgRequest of Type "put" with BodySize > 0 will be followed by a line containing a base64-encoded JSON string literal of the body.

type ProgResponse added in v0.7.2202 

type ProgResponse struct {
	ID  int64  // that corresponds to ProgRequest; they can be answered out of order
	Err string `json:",omitempty"` // if non-empty, the error

	// KnownCommands is included in the first message that cache helper program
	// writes to stdout on startup (with ID==0). It includes the
	// ProgRequest.Command types that are supported by the program.
	//
	// This lets us extend the protocol gracefully over time (adding "get2",
	// etc), or fail gracefully when needed. It also lets us verify the program
	// wants to be a cache helper.
	KnownCommands []ProgCmd `json:",omitempty"`

	Miss     bool       `json:",omitempty"` // cache miss
	OutputID []byte     `json:",omitempty"`
	Size     int64      `json:",omitempty"` // in bytes
	Time     *time.Time `json:",omitempty"` // an Entry.Time; when the object was added to the docs

	// DiskPath is the absolute path on disk of the ObjectID corresponding
	// a "get" request's ActionID (on cache hit) or a "put" request's
	// provided ObjectID.
	DiskPath string `json:",omitempty"`
}

ProgResponse is the JSON response from the child process to cmd/go.

With the exception of the first protocol message that the child writes to its stdout with ID==0 and KnownCommands populated, these are only sent in response to a ProgRequest from cmd/go.

ProgResponses can be sent in any order. The ID must match the request they're replying to.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.