proxycache

README

proxy_cache

缓存标准RFC

https://datatracker.ietf.org/doc/html/rfc7234 https://www.rfc-editor.org/rfc/rfc7234

缓存

proxy_cache_path

proxy_cache_path /data/nginx/cache levels=1:2;  # 第二层级有16*256=4096个目录
/data/nginx/cache/b/51/d7b6e5978e3f042f52e875005925e51b 

proxy_cache_path /data/nginx/cache levels=1:2:2 keys_zone=cache_zone:100m max_size=100g inactive=365d use_temp_path=off;

1. 实现目录缓存
2. 实现levels=1:2:2
3. 实现keys_zone=cache_zone:100m, name=cache_zone, 内存缓存CacheSizeMax=100m (默认使用btree实现)
4. 实现max_size=100g 最大文件系统缓存大小
5. inactive 指定了项目在不被访问的情况下能够在内存中保持的时间
6. use_temp_path=off or use_temp_path=/data/nginx/cache/temp
7. max_files 总缓存文件数量(与max_size共同决定能缓存的大小和数量)
8. 待定[manager_files=number] [manager_sleep=time] [manager_threshold=time] [loader_files=number] [loader_sleep=time] [loader_threshold=time] [purger=on|off] [purger_files=number] [purger_sleep=time] [purger_threshold=time];

参考

https://github.com/ydylla/fcache

fork from https://github.com/peterbourgon/diskv

vfs

https://github.com/rainycape/vfs

https://github.com/avfs/avfs

https://github.com/C2FO/vfs

https://github.com/spf13/afero

Documentation

Index

• type AdvancedTransformFunction
• type BTreeIndex
  • func (i *BTreeIndex) Delete(key string)
  • func (i *BTreeIndex) Initialize(less LessFunction, keys <-chan string)
  • func (i *BTreeIndex) Insert(key string)
  • func (i *BTreeIndex) Keys(from string, n int) []string
• type Compression
  • func NewGzipCompression() Compression
  • func NewGzipCompressionLevel(level int) Compression
  • func NewZlibCompression() Compression
  • func NewZlibCompressionLevel(level int) Compression
  • func NewZlibCompressionLevelDict(level int, dict []byte) Compression
• type Diskv
  • func New(o Options) *Diskv
  • func (d *Diskv) Erase(key string) error
  • func (d *Diskv) EraseAll() error
  • func (d *Diskv) Has(key string) bool
  • func (d *Diskv) Import(srcFilename, dstKey string, move bool) (err error)
  • func (d *Diskv) Keys(cancel <-chan struct{}) <-chan string
  • func (d *Diskv) KeysPrefix(prefix string, cancel <-chan struct{}) <-chan string
  • func (d *Diskv) Read(key string) ([]byte, error)
  • func (d *Diskv) ReadStream(key string, direct bool) (io.ReadCloser, error)
  • func (d *Diskv) ReadString(key string) string
  • func (d *Diskv) Write(key string, val []byte) error
  • func (d *Diskv) WriteStream(key string, r io.Reader, sync bool) error
  • func (d *Diskv) WriteString(key string, val string) error
• type Index
• type InverseTransformFunction
• type LessFunction
• type Options
• type PathKey
• type TransformFunction

Types

type AdvancedTransformFunction

type AdvancedTransformFunction func(s string) *PathKey

AdvancedTransformFunction transforms a key into a PathKey.

A PathKey contains a slice of strings, where each element in the slice represents a directory in the file path where the key's entry will eventually be stored, as well as the filename.

For example, if AdvancedTransformFunc transforms "abcdef/file.txt" to the PathKey {Path: ["ab", "cde", "f"], FileName: "file.txt"}, the final location of the data file will be <basedir>/ab/cde/f/file.txt.

You must provide an InverseTransformFunction if you use an AdvancedTransformFunction.

type BTreeIndex

type BTreeIndex struct {
	sync.RWMutex
	LessFunction
	*btree.BTree
}

BTreeIndex is an implementation of the Index interface using google/btree.

func (*BTreeIndex) Delete

func (i *BTreeIndex) Delete(key string)

Delete removes the given key (only) from the BTree tree.

func (*BTreeIndex) Initialize

func (i *BTreeIndex) Initialize(less LessFunction, keys <-chan string)

Initialize populates the BTree tree with data from the keys channel, according to the passed less function. It's destructive to the BTreeIndex.

func (*BTreeIndex) Insert

func (i *BTreeIndex) Insert(key string)

Insert inserts the given key (only) into the BTree tree.

func (*BTreeIndex) Keys

func (i *BTreeIndex) Keys(from string, n int) []string

Keys yields a maximum of n keys in order. If the passed 'from' key is empty, Keys will return the first n keys. If the passed 'from' key is non-empty, the first key in the returned slice will be the key that immediately follows the passed key, in key order.

type Compression

type Compression interface {
	Writer(dst io.Writer) (io.WriteCloser, error)
	Reader(src io.Reader) (io.ReadCloser, error)
}

Compression is an interface that Diskv uses to implement compression of data. Writer takes a destination io.Writer and returns a WriteCloser that compresses all data written through it. Reader takes a source io.Reader and returns a ReadCloser that decompresses all data read through it. You may define these methods on your own type, or use one of the NewCompression helpers.

func NewGzipCompression

func NewGzipCompression() Compression

NewGzipCompression returns a Gzip-based Compression.

func NewGzipCompressionLevel

func NewGzipCompressionLevel(level int) Compression

NewGzipCompressionLevel returns a Gzip-based Compression with the given level.

func NewZlibCompression

func NewZlibCompression() Compression

NewZlibCompression returns a Zlib-based Compression.

func NewZlibCompressionLevel

func NewZlibCompressionLevel(level int) Compression

NewZlibCompressionLevel returns a Zlib-based Compression with the given level.

func NewZlibCompressionLevelDict

func NewZlibCompressionLevelDict(level int, dict []byte) Compression

NewZlibCompressionLevelDict returns a Zlib-based Compression with the given level, based on the given dictionary.

type Diskv

type Diskv struct {
	Options
	// contains filtered or unexported fields
}

Diskv implements the Diskv interface. You shouldn't construct Diskv structures directly; instead, use the New constructor.

func New

func New(o Options) *Diskv

New returns an initialized Diskv structure, ready to use. If the path identified by baseDir already contains data, it will be accessible, but not yet cached.

func (*Diskv) Erase

func (d *Diskv) Erase(key string) error

Erase synchronously erases the given key from the disk and the cache.

func (*Diskv) EraseAll

func (d *Diskv) EraseAll() error

EraseAll will delete all of the data from the store, both in the cache and on the disk. Note that EraseAll doesn't distinguish diskv-related data from non- diskv-related data. Care should be taken to always specify a diskv base directory that is exclusively for diskv data.

func (*Diskv) Has

func (d *Diskv) Has(key string) bool

Has returns true if the given key exists.

func (*Diskv) Import

func (d *Diskv) Import(srcFilename, dstKey string, move bool) (err error)

Import imports the source file into diskv under the destination key. If the destination key already exists, it's overwritten. If move is true, the source file is removed after a successful import.

func (*Diskv) Keys

func (d *Diskv) Keys(cancel <-chan struct{}) <-chan string

Keys returns a channel that will yield every key accessible by the store, in undefined order. If a cancel channel is provided, closing it will terminate and close the keys channel.

func (*Diskv) KeysPrefix

func (d *Diskv) KeysPrefix(prefix string, cancel <-chan struct{}) <-chan string

KeysPrefix returns a channel that will yield every key accessible by the store with the given prefix, in undefined order. If a cancel channel is provided, closing it will terminate and close the keys channel. If the provided prefix is the empty string, all keys will be yielded.

func (*Diskv) Read

func (d *Diskv) Read(key string) ([]byte, error)

Read reads the key and returns the value. If the key is available in the cache, Read won't touch the disk. If the key is not in the cache, Read will have the side-effect of lazily caching the value.

func (*Diskv) ReadStream

func (d *Diskv) ReadStream(key string, direct bool) (io.ReadCloser, error)

ReadStream reads the key and returns the value (data) as an io.ReadCloser. If the value is cached from a previous read, and direct is false, ReadStream will use the cached value. Otherwise, it will return a handle to the file on disk, and cache the data on read.

If direct is true, ReadStream will lazily delete any cached value for the key, and return a direct handle to the file on disk.

If compression is enabled, ReadStream taps into the io.Reader stream prior to decompression, and caches the compressed data.

func (*Diskv) ReadString

func (d *Diskv) ReadString(key string) string

ReadString reads the key and returns a string value In case of error, an empty string is returned

func (*Diskv) Write

func (d *Diskv) Write(key string, val []byte) error

Write synchronously writes the key-value pair to disk, making it immediately available for reads. Write relies on the filesystem to perform an eventual sync to physical media. If you need stronger guarantees, see WriteStream.

func (*Diskv) WriteStream

func (d *Diskv) WriteStream(key string, r io.Reader, sync bool) error

WriteStream writes the data represented by the io.Reader to the disk, under the provided key. If sync is true, WriteStream performs an explicit sync on the file as soon as it's written.

bytes.Buffer provides io.Reader semantics for basic data types.

func (*Diskv) WriteString

func (d *Diskv) WriteString(key string, val string) error

WriteString writes a string key-value pair to disk

type Index

type Index interface {
	Initialize(less LessFunction, keys <-chan string)
	Insert(key string)
	Delete(key string)
	Keys(from string, n int) []string
}

Index is a generic interface for things that can provide an ordered list of keys.

type InverseTransformFunction

type InverseTransformFunction func(pathKey *PathKey) string

InverseTransformFunction takes a PathKey and converts it back to a Diskv key. In effect, it's the opposite of an AdvancedTransformFunction.

type LessFunction

type LessFunction func(string, string) bool

LessFunction is used to initialize an Index of keys in a specific order.

type Options

type Options struct {
	BasePath          string
	Transform         TransformFunction
	AdvancedTransform AdvancedTransformFunction
	InverseTransform  InverseTransformFunction
	CacheSizeMax      uint64 // bytes
	PathPerm          os.FileMode
	FilePerm          os.FileMode
	// If TempDir is set, it will enable filesystem atomic writes by
	// writing temporary files to that location before being moved
	// to BasePath.
	// Note that TempDir MUST be on the same device/partition as
	// BasePath.
	TempDir string

	Index     Index
	IndexLess LessFunction

	Compression Compression
}

Options define a set of properties that dictate Diskv behavior. All values are optional.

type PathKey

type PathKey struct {
	Path     []string
	FileName string
	// contains filtered or unexported fields
}

PathKey represents a string key that has been transformed to a directory and file name where the content will eventually be stored

type TransformFunction

type TransformFunction func(s string) []string

TransformFunction transforms a key into a slice of strings, with each element in the slice representing a directory in the file path where the key's entry will eventually be stored.

For example, if TransformFunc transforms "abcdef" to ["ab", "cde", "f"], the final location of the data file will be <basedir>/ab/cde/f/abcdef