filecache

README

Filecache

Implements a local, rudimentary file cache backed by an S3 bucket. This is useful for a file caching service.

Documentation

Index

• Constants
• Variables
• func DefaultExtension(ext string) option
• func DownloadTimeout(timeout time.Duration) option
• func DropboxDownload(dr *DownloadRecord, localFile io.Writer, downloadTimeout time.Duration) error
• func DropboxDownloader() option
• func S3Downloader(awsRegion string) option
• type DownloadManager
• type DownloadRecord
  • func NewDownloadRecord(url string, args map[string]string) (*DownloadRecord, error)
  • func (dr *DownloadRecord) GetUniqueName() string
• type FileCache
  • func New(size int, baseDir string, opts ...option) (*FileCache, error)
  • func (c *FileCache) Contains(dr *DownloadRecord) bool
  • func (c *FileCache) Fetch(dr *DownloadRecord) bool
  • func (c *FileCache) FetchNewerThan(dr *DownloadRecord, timestamp time.Time) bool
  • func (c *FileCache) GetFileName(dr *DownloadRecord) string
  • func (c *FileCache) MaybeDownload(dr *DownloadRecord) error
  • func (c *FileCache) Purge()
  • func (c *FileCache) PurgeAsync(doneChan chan struct{})
  • func (c *FileCache) Reload(dr *DownloadRecord) bool
• type RecordDownloaderFunc
• type S3RegionManagedDownloader
  • func NewS3RegionManagedDownloader(defaultRegion string) *S3RegionManagedDownloader
  • func (m *S3RegionManagedDownloader) Download(dr *DownloadRecord, localFile *os.File, downloadTimeout time.Duration) error
  • func (m *S3RegionManagedDownloader) GetDownloader(ctx context.Context, bucket string) (*s3manager.Downloader, error)

Constants

const (
	DownloadMangerS3 = iota
	DownloadMangerDropbox
)

Variables

var (

	// HashableArgs allows us to support various authentication headers in the future
	HashableArgs = map[string]struct{}{}
)

Functions

func DefaultExtension

func DefaultExtension(ext string) option

DefaultExtension sets the default extension which will be appended to cached files in the local directory

func DownloadTimeout

func DownloadTimeout(timeout time.Duration) option

DownloadTimeout sets the file download timeout

func DropboxDownload

func DropboxDownload(dr *DownloadRecord, localFile io.Writer, downloadTimeout time.Duration) error

DropboxDownload will download a file from the specified Dropbox location into localFile

func DropboxDownloader

func DropboxDownloader() option

DropboxDownloader allows the DownloadFunc to pull files from Dropbox accounts. Bubbles up errors from the Hashicrorp LRU library when something goes wrong there.

func S3Downloader

func S3Downloader(awsRegion string) option

S3Downloader allows the DownloadFunc to pull files from S3 buckets. Bucket names are passed at the first part of the path in files requested from the cache. Bubbles up errors from the Hashicrorp LRU library when something goes wrong there.

Types

type DownloadManager

type DownloadManager int

type DownloadRecord

type DownloadRecord struct {
	Manager    DownloadManager
	Path       string
	Args       map[string]string
	HashedArgs string
}

DownloadRecord contains information about a file which will be downloaded

func NewDownloadRecord

func NewDownloadRecord(url string, args map[string]string) (*DownloadRecord, error)

NewDownloadRecord converts the incoming URL path into a download record containing a cached filename (this is the filename on the backing store, not the cached filename locally) together with the args needed for authentication

func (*DownloadRecord) GetUniqueName

func (dr *DownloadRecord) GetUniqueName() string

GetUniqueName returns a *HOPEFULLY* unique name for the download record

type FileCache

type FileCache struct {
	BaseDir          string
	Cache            *lru.Cache
	Waiting          map[string]chan struct{}
	WaitLock         sync.Mutex
	DownloadFunc     func(dr *DownloadRecord, localPath string) error
	OnEvict          func(key interface{}, value interface{})
	DefaultExtension string
	DownloadTimeout  time.Duration
	// contains filtered or unexported fields
}

FileCache is a wrapper for hashicorp/golang-lru

func New

func New(size int, baseDir string, opts ...option) (*FileCache, error)

New returns a properly configured cache. Bubbles up errors from the Hashicrorp LRU library when something goes wrong there. The configured cache will have a noop DownloadFunc, which should be replaced if you want to actually get files from somewhere. Or, look at NewS3Cache() which is backed by Amazon S3.

func (*FileCache) Contains

func (c *FileCache) Contains(dr *DownloadRecord) bool

Contains looks to see if we have an entry in the cache for this file.

func (*FileCache) Fetch

func (c *FileCache) Fetch(dr *DownloadRecord) bool

Fetch will return true if we have the file, or will go download the file and return true if we can. It will return false only if it's unable to fetch the file from the backing store (S3).

func (*FileCache) FetchNewerThan

func (c *FileCache) FetchNewerThan(dr *DownloadRecord, timestamp time.Time) bool

FetchNewerThan will look in the cache for a file, make sure it's newer than timestamp, and if so return true. Otherwise it will possibly download the file and only return false if it's unable to do so.

func (*FileCache) GetFileName

func (c *FileCache) GetFileName(dr *DownloadRecord) string

GetFileName returns the full storage path and file name for a file, if it were in the cache. This does _not_ check to see if the file is actually _in_ the cache. This builds a cache structure of up to 256 directories, each beginning with the first 2 letters of the FNV32 hash of the filename. This is then joined to the base dir and MD5 hashed filename to form the cache path for each file. It preserves the file extension (if present)

e.g. /base_dir/2b/b0804ec967f48520697662a204f5fe72

func (*FileCache) MaybeDownload

func (c *FileCache) MaybeDownload(dr *DownloadRecord) error

MaybeDownload might go out to the backing store (S3) and get the file if the file isn't already being downloaded in another routine. In both cases it will block until the download is completed either by this goroutine or another one.

func (*FileCache) Purge

func (c *FileCache) Purge()

Purge clears all the files from the cache (via the onEvict callback for each key).

func (*FileCache) PurgeAsync

func (c *FileCache) PurgeAsync(doneChan chan struct{})

PurgeAsync clears all the files from the cache and takes an optional channel to close when the purge has completed.

func (*FileCache) Reload

func (c *FileCache) Reload(dr *DownloadRecord) bool

Reload will remove a file from the cache and attempt to reload from the backing store, calling MaybeDownload().

type RecordDownloaderFunc

type RecordDownloaderFunc = func(dr *DownloadRecord, localFile *os.File) error

type S3RegionManagedDownloader

type S3RegionManagedDownloader struct {
	sync.RWMutex
	DefaultRegion   string
	DownloaderCache map[string]*s3manager.Downloader // Map buckets to regions
}

Manages a cache of s3manager.Downloader s that have been configured for their correct region.

func NewS3RegionManagedDownloader

func NewS3RegionManagedDownloader(defaultRegion string) *S3RegionManagedDownloader

NewS3RegionManagedDownloader returns a configured instance where the default bucket region will be as passed. This means bucket lookups from the cache will prefer that region when hinting to S3 which region they believe a bucket lives in.

func (*S3RegionManagedDownloader) Download

func (m *S3RegionManagedDownloader) Download(dr *DownloadRecord, localFile *os.File, downloadTimeout time.Duration) error

Download will download a file from the specified S3 bucket into localFile

func (*S3RegionManagedDownloader) GetDownloader

func (m *S3RegionManagedDownloader) GetDownloader(ctx context.Context, bucket string) (*s3manager.Downloader, error)

GetDownloader looks up a bucket in the cache and returns a configured s3manager.Downloader for it or provisions a new one and returns that. NOTE! This is never flushed and so should not be used with an unlimited number of buckets! The first few requests will incur an additional penalty of roundtrips to Amazon to look up the region fo the requested S3 bucket.