sharedcache

Documentation

Index

• Variables
• type Cache
  • func Open(fs vfs.FS, logger base.Logger, fsDir string, blockSize int, ...) (*Cache, error)
  • func (c *Cache) Close() error
  • func (c *Cache) Metrics() Metrics
  • func (c *Cache) ReadAt(ctx context.Context, fileNum base.DiskFileNum, p []byte, ofs int64, ...) error
• type Metrics
• type ReadFlags

Variables

var (
	IOBuckets           = prometheus.ExponentialBucketsRange(float64(time.Millisecond*1), float64(10*time.Second), 50)
	ChannelWriteBuckets = prometheus.ExponentialBucketsRange(float64(time.Microsecond*1), float64(10*time.Second), 50)
)

Exported to enable exporting from package pebble to enable exporting metrics with below buckets in CRDB.

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a persistent cache backed by a local filesystem. It is intended to cache data that is in slower shared storage (e.g. S3), hence the package name 'sharedcache'.

func Open

func Open(
	fs vfs.FS,
	logger base.Logger,
	fsDir string,
	blockSize int,

	shardingBlockSize int64,
	sizeBytes int64,
	numShards int,
) (*Cache, error)

Open opens a cache. If there is no existing cache at fsDir, a new one is created.

func (*Cache) Close

func (c *Cache) Close() error

Close closes the cache. Methods such as ReadAt should not be called after Close is called.

func (*Cache) Metrics

func (c *Cache) Metrics() Metrics

Metrics return metrics for the cache. Callers should not mutate the returned histograms, which are pointer types.

func (*Cache) ReadAt

func (c *Cache) ReadAt(
	ctx context.Context,
	fileNum base.DiskFileNum,
	p []byte,
	ofs int64,
	objReader remote.ObjectReader,
	objSize int64,
	flags ReadFlags,
) error

ReadAt performs a read form an object, attempting to use cached data when possible.

type Metrics

type Metrics struct {
	// The number of sstable bytes stored in the cache.
	Size int64
	// The count of cache blocks in the cache (not sstable blocks).
	Count int64

	// The number of calls to ReadAt.
	TotalReads int64
	// The number of calls to ReadAt that require reading data from 2+ shards.
	MultiShardReads int64
	// The number of calls to ReadAt that require reading data from 2+ cache blocks.
	MultiBlockReads int64
	// The number of calls to ReadAt where all data returned was read from the cache.
	ReadsWithFullHit int64
	// The number of calls to ReadAt where some data returned was read from the cache.
	ReadsWithPartialHit int64
	// The number of calls to ReadAt where no data returned was read from the cache.
	ReadsWithNoHit int64

	// The number of times a cache block was evicted from the cache.
	Evictions int64
	// The number of times writing a cache block to the cache failed.
	WriteBackFailures int64

	// The latency of calls to get some data from the cache.
	GetLatency prometheus.Histogram
	// The latency of reads of a single cache block from disk.
	DiskReadLatency prometheus.Histogram
	// The latency of writing data to write back to the cache to a channel.
	// Generally should be low, but if the channel is full, could be high.
	QueuePutLatency prometheus.Histogram
	// The latency of calls to put some data read from block storage into the cache.
	PutLatency prometheus.Histogram
	// The latency of writes of a single cache block to disk.
	DiskWriteLatency prometheus.Histogram
}

Metrics is a struct containing metrics exported by the secondary cache. TODO(josh): Reconsider the set of metrics exported by the secondary cache before we release the secondary cache to users. We choose to export many metrics right now, so we learn a lot from the benchmarking we are doing over the 23.2 cycle.

type ReadFlags

type ReadFlags struct {
	// ReadOnly instructs ReadAt to not write any new data into the cache; it is
	// used when the data is unlikely to be used again.
	ReadOnly bool
}

ReadFlags contains options for Cache.ReadAt.