cache

Documentation

Overview

Package cache provides helpers for building a caching system based on io/fs.FS.

Cache libraries implement cache.CacheFS and can be used to create a cache.FS.

A call to FS.ReadFile() will read from the cache and on a cache miss will read from storage. If storage has the file, the file is loaded into the cache.

FS also implements CacheFS, so we can build muli-level caches such as an in-memory cache that pulls from a disk cache which pulls from Redis which pulls from Azure Blob Storage in a waterfall like system.

Create our FS that uses long term storage (Azure Blob Storage):

cred, err := msi.Token(msi.AppID{ID: "your app ID"})
if err != nil {
	panic(err)
}

cloudStore, err := blob.NewFS("account", "container", *cred)
if err != nil {
	// Do something
}

Create a Redis CacheFS:

	redisFS := redis.New(
		redis.Args{
			Addr:     "localhost:6379",
        		Password: "", // no password set
        		DB:       0,  // use default DB
		},
	)

Setup our first cache layer which tries redis and then fills from a blob:

networkCache, err := cache.New(redisFS, cloudStore)
if err != nil {
	// Do something
}

Setup our local disk FS:

diskFS, err := disk.New("")
if err != nil {
	// Do something
}

Create our second cache layer, pulls from disk first, then redis, then blob:

diskCache, err := cache.New(diskFS, networkCache)
if err != nil {
	// Do something
}

Create our memory cache;

memCache, err := memfs.New()
if err != nil {
	// Do something
}

Create our final cache, pulls from memory, then disk, then redis, then blob:

cacheSys, err := cache.New(memCache, diskCache)
if err != nil {
	// Do something
}

Get a file from our cache:

// This first attempts to read this from memory. If it doesn't exist, it
// attempts to grab from our disk. If it doesn't exist, it tries to
// read from Redis. If it doesn't exist, it reads it from Azure blob storage.
// Once the file is found, we backfill each layer. This works best when
// each layer down holds the data for longer than the previous layer until
// you reach permanent storage.
b, err := cacheSys.ReadFile("/path/to/file")
if err != nil {
	// Do something
}

Index

• type CacheFS
• type FS
  • func New(cache CacheFS, store CacheFS) (*FS, error)
  • func (f *FS) Open(name string) (fs.File, error)
  • func (f *FS) OpenFile(name string, perms fs.FileMode, options ...jsfs.OFOption) (fs.File, error)
  • func (f *FS) ReadFile(name string) ([]byte, error)
  • func (f *FS) Stat(name string) (fs.FileInfo, error)
  • func (f *FS) WriteFile(name string, content []byte, perm fs.FileMode) error
• type SetFiller

Types

type CacheFS

type CacheFS interface {
	jsfs.Writer
	fs.ReadFileFS
	fs.StatFS
}

CacheFS represents some cache that we can read and write files from.

type FS

type FS struct {

	// Log allows you to replace the standard logger to one of
	// your own choosing. By default this logs to Stderr.
	Log jsfs.Logger

	// FilledBy indicates what cache layer filled the request of a ReadFile().
	// This is only set during testing and exists due to the lack of Context on
	// the interfaces.
	FilledBy string
	// contains filtered or unexported fields
}

FS implemenents io/fs.FS to provide a cache reader and writer.

func New

func New(cache CacheFS, store CacheFS) (*FS, error)

New is the constructor for FS.

func (*FS) Open

func (f *FS) Open(name string) (fs.File, error)

Open opens a file for reading. The file will be served out of cache to start and if not available it will be served out of storage. Using Open() does NOT cause a non-cached file to be cache.

func (*FS) OpenFile

func (f *FS) OpenFile(name string, perms fs.FileMode, options ...jsfs.OFOption) (fs.File, error)

OpenFile implements fs.OpenFiler.OpenFile(). This pulls from the storage FS and therefore you pass options for that FS. This is here for completeness, but ReadFile() and WriteFile() are what generally you should use unless you want to access storage and bypass the cache layer completely.

func (*FS) ReadFile

func (f *FS) ReadFile(name string) ([]byte, error)

ReadFile reads a file. This checks the cache first and then checks storage. If the file is found in storage, a call to the cache's WriteFile() is made in a separate go routine so that it is served out of cache in the future.

func (*FS) Stat

func (f *FS) Stat(name string) (fs.FileInfo, error)

Stat implememnts fs.StatFS.Stat().

func (*FS) WriteFile

func (f *FS) WriteFile(name string, content []byte, perm fs.FileMode) error

WriteFile implememnts jsfs.Writer.WriteFile().

type SetFiller

type SetFiller interface {
	SetFiller(fsys CacheFS)
}

SetFiller provides a function for setting a jsfs.Writer implementaiton that does cache fills on misses. Some CacheFS implementation need this because they support automatic cache fill mechanisms instead of just Getter()/Setter() methods.