diskcache

Documentation

Overview

Package diskcache implements an on-disk cache of a remote file tree.

A cache, of type *Cache, manages a local directory holding cached copies of remote files obtained from a loader, of type Loader. The loader handles reading remote files and checking whether a cached copy is still valid.

To open a file, the cache first searches the local directory for a cached copy. After being revalidated or fetched by the loader, a copy is considered to be valid for a fixed duration, after which it is kept but considered expired. If a copy exists and has not expired, the cache uses it. If no copy exists or the copy has expired, the cache invokes the loader to refresh the copy or fetch a new copy.

A cached file is checked for expiration only during open, not during individual reads.

A cache may be used by multiple goroutines simultaneously. Furthermore, multiple caches, even in separate processes, may share a single directory, provided the caches use the same loader. In this way, a cache can be reused across multiple program executions and also be shared by concurrently executing programs. If a cache directory is to be shared by concurrent programs, it should usually be on local disk, because Unix file locking over network file systems is almost always broken.

There is no cache for file load errors.

On-Disk Format

Each cached file stored on disk using a name derived from the SHA1 hash of the file name. The first three hex digits name a subdirectory of the cache root directory, and the remaining seventeen digits are used as the base name of a group of files within that subdirectory:

123/45678901234567890.data
123/45678901234567890.meta
123/45678901234567890.used
123/45678901234567890.next

The .data file is the cached file content. If it exists, it is a complete copy, never a partial one.

The .meta file is the metadata associated with the .data file. It contains the JSON encoding of a metadata struct. The modification time of the .meta file is the time that the .data file was last downloaded or revalidated. The .data file is considered to be valid until that time plus the expiration period. As a special case, if the .meta file has a modification time of January 1, 1970 00:00:00 UTC (Unix time 0), the .data file is considered expired, even if there is no expiration period.

The .used file holds a single \n byte. It is rewritten each time the .data file is opened to satisfy a file open operation. The modification time of the .used file is therefore the time of the last use of the file.

The .next file holds the next version of the cached file, while it is being downloaded. Once the download has completed, the cache renames the .next file onto the .data file. This sequence avoids overwriting the content of the .data file, which other clients might still be reading.

To allow multiple instances of a cache to manage a shared directory, if a cache is doing the initial download of a file or revalidating an expired copy or redownloading a new copy, it must hold an exclusive BSD file lock on the .meta file (using flock(2)).

After downloading a new file and installing it as a .data file, the cache must check that it has not exceeded the on-disk size limit. It checks by reading the sizes of all the .data files in the directory tree and the modification times of the .used files. It then removes the oldest cached files (.data, .meta, and .used) until the data files again fit within the limit. To remove a file, the cache must hold the .meta file lock.

Warning Warning Warning

This package is unfinished. In particular, DeleteAll and ExpireAll are unimplemented, as is the code to delete files to stay within the maximum data size limit.

Index

• type Cache
  • func New(dir string, loader Loader) (*Cache, error)
  • func (c *Cache) Delete(path string) error
  • func (c *Cache) DeleteAll() error
  • func (c *Cache) Expire(path string) error
  • func (c *Cache) ExpireAll() error
  • func (c *Cache) Open(path string) (*os.File, error)
  • func (c *Cache) ReadFile(path string) ([]byte, error)
  • func (c *Cache) SetExpiration(d time.Duration)
  • func (c *Cache) SetMaxData(max int64)
• type Loader

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

A Cache provides read-only access to a remote file tree, caching opened files on local disk.

func New

func New(dir string, loader Loader) (*Cache, error)

New returns a new Cache that reads files from loader, caching at most max bytes in the directory dir. If dir does not exist, New will attempt to create it.

func (*Cache) Delete

func (c *Cache) Delete(path string) error

Delete deletes the cache entry for the file with the given path.

func (*Cache) DeleteAll

func (c *Cache) DeleteAll() error

DeleteAll deletes all the cache entries.

func (*Cache) Expire

func (c *Cache) Expire(path string) error

Expire marks the cache entry for the file with the given path as expired. The cache will have to revalidate the local copy, if any, before using it again.

func (*Cache) ExpireAll

func (c *Cache) ExpireAll() error

ExpireAll marks all cache entries as expired.

func (*Cache) Open

func (c *Cache) Open(path string) (*os.File, error)

Open opens the file with the given path. The caller is responsible for closing the returned file when finished with it. The elements in a file path are separated by slash ('/', U+002F) characters, regardless of host operating system convention.

func (*Cache) ReadFile

func (c *Cache) ReadFile(path string) ([]byte, error)

func (*Cache) SetExpiration

func (c *Cache) SetExpiration(d time.Duration)

SetExpiration sets the duration after which a cached copy is considered to have expired. If the duration d is zero (the default), cached copies never expire.

func (*Cache) SetMaxData

func (c *Cache) SetMaxData(max int64)

SetMaxData sets the maximum bytes of data to hold in cached copies. The limit is imposed in a best effort fashion. In particular, it does not apply to old copies that have not yet been closed, nor to new copies that have not finished downloading, nor to cache metadata.

type Loader

type Loader interface {
	Load(path string, target *os.File, meta []byte) (cacheValid bool, newMeta []byte, err error)
}

Loader is the interface Cache uses to load remote file content.

The Load method fetches path from the remote location, writing it to target. If the cache already has a (possibly expired) copy of the file, meta will be the metadata returned by a previous call to Load. Otherwise meta is nil. If the cached copy is still valid, Load should return cacheValid==true, newMeta==meta (or an updated version), and err==nil. Otherwise, Load should fetch the data, write it to target, and return cacheValid==true, a new metadata block in newMeta, and err==nil.

The elements in a file path are separated by slash ('/', U+002F) characters, regardless of host operating system convention.