storecache

package
v0.24.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 9, 2021 License: Apache-2.0 Imports: 32 Imported by: 5

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInvalidBucketCacheKeyFormat = errors.New("key has invalid format")
	ErrInvalidBucketCacheKeyVerb   = errors.New("key has invalid verb")
	ErrParseKeyInt                 = errors.New("failed to parse integer in key")
)
View Source 
var (
	DefaultInMemoryIndexCacheConfig = InMemoryIndexCacheConfig{
		MaxSize:     250 * 1024 * 1024,
		MaxItemSize: 125 * 1024 * 1024,
	}
)

Functions

func IsValidVerb added in v0.24.0 

func IsValidVerb(v VerbType) bool

IsValidVerb checks if the VerbType matches the predefined verbs.

func NewCachingBucketFromYaml added in v0.13.0 

func NewCachingBucketFromYaml(yamlContent []byte, bucket objstore.Bucket, logger log.Logger, reg prometheus.Registerer) (objstore.InstrumentedBucket, error)

NewCachingBucketFromYaml uses YAML configuration to create new caching bucket.

Types

type BucketCacheKey added in v0.24.0 

type BucketCacheKey struct {
	Verb  VerbType
	Name  string
	Start int64
	End   int64
}

func ParseBucketCacheKey added in v0.24.0 

func ParseBucketCacheKey(key string) (BucketCacheKey, error)

ParseBucketCacheKey parses a string and returns BucketCacheKey.

func (BucketCacheKey) String added in v0.24.0 

func (ck BucketCacheKey) String() string

String returns the string representation of BucketCacheKey.

type BucketCacheProvider added in v0.13.0 

type BucketCacheProvider string

BucketCacheProvider is a type used to evaluate all bucket cache providers. 

const (
	InMemoryBucketCacheProvider  BucketCacheProvider = "IN-MEMORY" // In-memory cache-provider for caching bucket.
	MemcachedBucketCacheProvider BucketCacheProvider = "MEMCACHED" // Memcached cache-provider for caching bucket.
)

type CachingBucket added in v0.13.0 

type CachingBucket struct {
	objstore.Bucket
	// contains filtered or unexported fields
}

CachingBucket implementation that provides some caching features, based on passed configuration.

func NewCachingBucket added in v0.13.0 

func NewCachingBucket(b objstore.Bucket, cfg *CachingBucketConfig, logger log.Logger, reg prometheus.Registerer) (*CachingBucket, error)

NewCachingBucket creates new caching bucket with provided configuration. Configuration should not be changed after creating caching bucket.

func (*CachingBucket) Attributes added in v0.14.0 

func (cb *CachingBucket) Attributes(ctx context.Context, name string) (objstore.ObjectAttributes, error)

func (*CachingBucket) Exists added in v0.13.0 

func (cb *CachingBucket) Exists(ctx context.Context, name string) (bool, error)

func (*CachingBucket) Get added in v0.13.0 

func (cb *CachingBucket) Get(ctx context.Context, name string) (io.ReadCloser, error)

func (*CachingBucket) GetRange added in v0.13.0 

func (cb *CachingBucket) GetRange(ctx context.Context, name string, off, length int64) (io.ReadCloser, error)

func (*CachingBucket) IsObjNotFoundErr added in v0.13.0 

func (cb *CachingBucket) IsObjNotFoundErr(err error) bool

func (*CachingBucket) Iter added in v0.13.0 

func (cb *CachingBucket) Iter(ctx context.Context, dir string, f func(string) error, options ...objstore.IterOption) error

func (*CachingBucket) Name added in v0.13.0 

func (cb *CachingBucket) Name() string

func (*CachingBucket) ReaderWithExpectedErrs added in v0.13.0 

func (cb *CachingBucket) ReaderWithExpectedErrs(expectedFunc objstore.IsOpFailureExpectedFunc) objstore.BucketReader

func (*CachingBucket) WithExpectedErrs added in v0.13.0 

func (cb *CachingBucket) WithExpectedErrs(expectedFunc objstore.IsOpFailureExpectedFunc) objstore.Bucket

type CachingBucketConfig added in v0.13.0 

type CachingBucketConfig struct {
	// contains filtered or unexported fields
}

CachingBucketConfig contains low-level configuration for individual bucket operations. This is not exposed to the user, but it is expected that code sets up individual operations based on user-provided configuration.

func NewCachingBucketConfig added in v0.13.0 

func NewCachingBucketConfig() *CachingBucketConfig

func (*CachingBucketConfig) CacheAttributes added in v0.14.0 

func (cfg *CachingBucketConfig) CacheAttributes(configName string, cache cache.Cache, matcher func(name string) bool, ttl time.Duration)

CacheAttributes configures caching of "Attributes" operation for matching files.

func (*CachingBucketConfig) CacheExists added in v0.13.0 

func (cfg *CachingBucketConfig) CacheExists(configName string, cache cache.Cache, matcher func(string) bool, existsTTL, doesntExistTTL time.Duration)

CacheExists configures caching of "Exists" operation for matching files. Negative values are cached as well.

func (*CachingBucketConfig) CacheGet added in v0.13.0 

func (cfg *CachingBucketConfig) CacheGet(configName string, cache cache.Cache, matcher func(string) bool, maxCacheableSize int, contentTTL, existsTTL, doesntExistTTL time.Duration)

CacheGet configures caching of "Get" operation for matching files. Content of the object is cached, as well as whether object exists or not.

func (*CachingBucketConfig) CacheGetRange added in v0.13.0 

func (cfg *CachingBucketConfig) CacheGetRange(configName string, cache cache.Cache, matcher func(string) bool, subrangeSize int64, attributesTTL, subrangeTTL time.Duration, maxSubRequests int)

CacheGetRange configures caching of "GetRange" operation. Subranges (aligned on subrange size) are cached individually. Since caching operation needs to know the object size to compute correct subranges, object size is cached as well. Single "GetRange" requests can result in multiple smaller GetRange sub-requests issued on the underlying bucket. MaxSubRequests specifies how many such subrequests may be issued. Values <= 0 mean there is no limit (requests for adjacent missing subranges are still merged).

func (*CachingBucketConfig) CacheIter added in v0.13.0 

func (cfg *CachingBucketConfig) CacheIter(configName string, cache cache.Cache, matcher func(string) bool, ttl time.Duration, codec IterCodec)

CacheIter configures caching of "Iter" operation for matching directories.

type CachingWithBackendConfig added in v0.13.0 

type CachingWithBackendConfig struct {
	Type          BucketCacheProvider `yaml:"type"`
	BackendConfig interface{}         `yaml:"config"`

	// Basic unit used to cache chunks.
	ChunkSubrangeSize int64 `yaml:"chunk_subrange_size"`

	// Maximum number of GetRange requests issued by this bucket for single GetRange call. Zero or negative value = unlimited.
	MaxChunksGetRangeRequests int `yaml:"max_chunks_get_range_requests"`

	// TTLs for various cache items.
	ChunkObjectAttrsTTL time.Duration `yaml:"chunk_object_attrs_ttl"`
	ChunkSubrangeTTL    time.Duration `yaml:"chunk_subrange_ttl"`

	// How long to cache result of Iter call in root directory.
	BlocksIterTTL time.Duration `yaml:"blocks_iter_ttl"`

	// Config for Exists and Get operations for metadata files.
	MetafileExistsTTL      time.Duration `yaml:"metafile_exists_ttl"`
	MetafileDoesntExistTTL time.Duration `yaml:"metafile_doesnt_exist_ttl"`
	MetafileContentTTL     time.Duration `yaml:"metafile_content_ttl"`
	MetafileMaxSize        model.Bytes   `yaml:"metafile_max_size"`
}

CachingWithBackendConfig is a configuration of caching bucket used by Store component.

func (*CachingWithBackendConfig) Defaults added in v0.13.0 

func (cfg *CachingWithBackendConfig) Defaults()

type InMemoryIndexCache added in v0.10.0 

type InMemoryIndexCache struct {
	// contains filtered or unexported fields
}

func NewInMemoryIndexCache added in v0.10.0 

func NewInMemoryIndexCache(logger log.Logger, reg prometheus.Registerer, conf []byte) (*InMemoryIndexCache, error)

NewInMemoryIndexCache creates a new thread-safe LRU cache for index entries and ensures the total cache size approximately does not exceed maxBytes.

func NewInMemoryIndexCacheWithConfig added in v0.10.0 

func NewInMemoryIndexCacheWithConfig(logger log.Logger, reg prometheus.Registerer, config InMemoryIndexCacheConfig) (*InMemoryIndexCache, error)

NewInMemoryIndexCacheWithConfig creates a new thread-safe LRU cache for index entries and ensures the total cache size approximately does not exceed maxBytes.

func (*InMemoryIndexCache) FetchMultiPostings added in v0.10.0 

func (c *InMemoryIndexCache) FetchMultiPostings(_ context.Context, blockID ulid.ULID, keys []labels.Label) (hits map[labels.Label][]byte, misses []labels.Label)

FetchMultiPostings fetches multiple postings - each identified by a label - and returns a map containing cache hits, along with a list of missing keys.

func (*InMemoryIndexCache) FetchMultiSeries added in v0.10.0 

func (c *InMemoryIndexCache) FetchMultiSeries(_ context.Context, blockID ulid.ULID, ids []storage.SeriesRef) (hits map[storage.SeriesRef][]byte, misses []storage.SeriesRef)

FetchMultiSeries fetches multiple series - each identified by ID - from the cache and returns a map containing cache hits, along with a list of missing IDs.

func (*InMemoryIndexCache) StorePostings added in v0.10.0 

func (c *InMemoryIndexCache) StorePostings(_ context.Context, blockID ulid.ULID, l labels.Label, v []byte)

StorePostings sets the postings identified by the ulid and label to the value v, if the postings already exists in the cache it is not mutated.

func (*InMemoryIndexCache) StoreSeries added in v0.10.0 

func (c *InMemoryIndexCache) StoreSeries(_ context.Context, blockID ulid.ULID, id storage.SeriesRef, v []byte)

StoreSeries sets the series identified by the ulid and id to the value v, if the series already exists in the cache it is not mutated.

type InMemoryIndexCacheConfig added in v0.10.0 

type InMemoryIndexCacheConfig struct {
	// MaxSize represents overall maximum number of bytes cache can contain.
	MaxSize model.Bytes `yaml:"max_size"`
	// MaxItemSize represents maximum size of single item.
	MaxItemSize model.Bytes `yaml:"max_item_size"`
}

InMemoryIndexCacheConfig holds the in-memory index cache config.

type IndexCache 

type IndexCache interface {
	// StorePostings stores postings for a single series.
	StorePostings(ctx context.Context, blockID ulid.ULID, l labels.Label, v []byte)

	// FetchMultiPostings fetches multiple postings - each identified by a label -
	// and returns a map containing cache hits, along with a list of missing keys.
	FetchMultiPostings(ctx context.Context, blockID ulid.ULID, keys []labels.Label) (hits map[labels.Label][]byte, misses []labels.Label)

	// StoreSeries stores a single series.
	StoreSeries(ctx context.Context, blockID ulid.ULID, id storage.SeriesRef, v []byte)

	// FetchMultiSeries fetches multiple series - each identified by ID - from the cache
	// and returns a map containing cache hits, along with a list of missing IDs.
	FetchMultiSeries(ctx context.Context, blockID ulid.ULID, ids []storage.SeriesRef) (hits map[storage.SeriesRef][]byte, misses []storage.SeriesRef)
}

IndexCache is the interface exported by index cache backends.

func NewIndexCache 

func NewIndexCache(logger log.Logger, confContentYaml []byte, reg prometheus.Registerer) (IndexCache, error)

NewIndexCache initializes and returns new index cache.

type IndexCacheConfig added in v0.10.0 

type IndexCacheConfig struct {
	Type   IndexCacheProvider `yaml:"type"`
	Config interface{}        `yaml:"config"`
}

IndexCacheConfig specifies the index cache config.

type IndexCacheProvider added in v0.10.0 

type IndexCacheProvider string
const (
	INMEMORY  IndexCacheProvider = "IN-MEMORY"
	MEMCACHED IndexCacheProvider = "MEMCACHED"
)

type IterCodec added in v0.13.0 

type IterCodec interface {
	Encode(files []string) ([]byte, error)
	Decode(cachedData []byte) ([]string, error)
}

Codec for encoding and decoding results of Iter call.

type JSONIterCodec added in v0.13.0 

type JSONIterCodec struct{}

JSONIterCodec encodes iter results into JSON. Suitable for root dir.

func (JSONIterCodec) Decode added in v0.13.0 

func (jic JSONIterCodec) Decode(data []byte) ([]string, error)

func (JSONIterCodec) Encode added in v0.13.0 

func (jic JSONIterCodec) Encode(files []string) ([]byte, error)

type MemcachedIndexCache added in v0.10.0 

type MemcachedIndexCache struct {
	// contains filtered or unexported fields
}

MemcachedIndexCache is a memcached-based index cache.

func NewMemcachedIndexCache added in v0.10.0 

func NewMemcachedIndexCache(logger log.Logger, memcached cacheutil.MemcachedClient, reg prometheus.Registerer) (*MemcachedIndexCache, error)

NewMemcachedIndexCache makes a new MemcachedIndexCache.

func (*MemcachedIndexCache) FetchMultiPostings added in v0.10.0 

func (c *MemcachedIndexCache) FetchMultiPostings(ctx context.Context, blockID ulid.ULID, lbls []labels.Label) (hits map[labels.Label][]byte, misses []labels.Label)

FetchMultiPostings fetches multiple postings - each identified by a label - and returns a map containing cache hits, along with a list of missing keys. In case of error, it logs and return an empty cache hits map.

func (*MemcachedIndexCache) FetchMultiSeries added in v0.10.0 

func (c *MemcachedIndexCache) FetchMultiSeries(ctx context.Context, blockID ulid.ULID, ids []storage.SeriesRef) (hits map[storage.SeriesRef][]byte, misses []storage.SeriesRef)

FetchMultiSeries fetches multiple series - each identified by ID - from the cache and returns a map containing cache hits, along with a list of missing IDs. In case of error, it logs and return an empty cache hits map.

func (*MemcachedIndexCache) StorePostings added in v0.10.0 

func (c *MemcachedIndexCache) StorePostings(ctx context.Context, blockID ulid.ULID, l labels.Label, v []byte)

StorePostings sets the postings identified by the ulid and label to the value v. The function enqueues the request and returns immediately: the entry will be asynchronously stored in the cache.

func (*MemcachedIndexCache) StoreSeries added in v0.10.0 

func (c *MemcachedIndexCache) StoreSeries(ctx context.Context, blockID ulid.ULID, id storage.SeriesRef, v []byte)

StoreSeries sets the series identified by the ulid and id to the value v. The function enqueues the request and returns immediately: the entry will be asynchronously stored in the cache.

type VerbType added in v0.24.0 

type VerbType string

VerbType is the type of operation whose result has been stored in the caching bucket's cache. 

const (
	ExistsVerb     VerbType = "exists"
	ContentVerb    VerbType = "content"
	IterVerb       VerbType = "iter"
	AttributesVerb VerbType = "attrs"
	SubrangeVerb   VerbType = "subrange"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.