sharding

Documentation

Index

• Constants
• Variables
• func GuessShardFactor(bytes, maxBytesPerShard uint64, maxShards int) int
• func LinearShards(n int, bytes uint64) []logproto.Shard
• type ForSeries
• type ForSeriesFunc
  • func (f ForSeriesFunc) ForSeries(ctx context.Context, userID string, fpFilter index.FingerprintFilter, ...) error
• type PowerOfTwoSharding
  • func (p PowerOfTwoSharding) ShardsFor(bytes uint64, maxBytesPerShard uint64) []logproto.Shard
• type SizedFP
• type SizedFPs
  • func (xs SizedFPs) Len() int
  • func (xs SizedFPs) Less(i, j int) bool
  • func (xs SizedFPs) ShardsFor(targetShardBytes uint64) (res []logproto.Shard)
  • func (xs SizedFPs) Swap(i, j int)

Constants

const (
	DefaultTSDBMaxBytesPerShard = 600 << 20 // 600MB
)

Variables

var (
	SizedFPsPool = queue.NewSlicePool[SizedFP](1<<8, 1<<16, 4) // 256->65536
)

Functions

func GuessShardFactor

func GuessShardFactor(bytes, maxBytesPerShard uint64, maxShards int) int

Since we shard by powers of two and we increase shard factor once each shard surpasses maxBytesPerShard, if the shard factor is at least two, the range of data per shard is (maxBytesPerShard/2, maxBytesPerShard] For instance, for a maxBytesPerShard of 500MB and a query touching 1000MB, we split into two shards of 500MB. If there are 1004MB, we split into four shards of 251MB.

func LinearShards

func LinearShards(n int, bytes uint64) []logproto.Shard

LinearShards is a sharding implementation that splits the data into equal sized shards covering the entire keyspace. It populates the `bytes` of each shard's stats with a proportional estimation

Types

type ForSeries

type ForSeries interface {
	ForSeries(
		ctx context.Context,
		userID string,
		fpFilter index.FingerprintFilter,
		from model.Time,
		through model.Time,
		fn func(
			labels.Labels,
			model.Fingerprint,
			[]index.ChunkMeta,
		) (stop bool),
		matchers ...*labels.Matcher,
	) error
}

General purpose iteration over series. Makes it easier to build custom functionality on top of indices of different types without them all implementing the same feature. The passed callback must _not_ capture its arguments. They're reused for each call for performance. The passed callback may be executed concurrently, so any shared state must be protected by the caller. NB: This is a low-level API and should be used with caution. NB: It's possible for the callback to be called multiple times for the same series but possibly different chunks, such as when the Index is backed by multiple files with the same series present. NB(owen-d): mainly in this package to avoid circular dependencies elsewhere

type ForSeriesFunc

type ForSeriesFunc func(
	ctx context.Context,
	userID string,
	fpFilter index.FingerprintFilter,
	from model.Time,
	through model.Time,
	fn func(
		labels.Labels,
		model.Fingerprint,
		[]index.ChunkMeta,
	) (stop bool),
	matchers ...*labels.Matcher,
) error

function Adapter for ForSeries implementation

func (ForSeriesFunc) ForSeries

func (f ForSeriesFunc) ForSeries(
	ctx context.Context,
	userID string,
	fpFilter index.FingerprintFilter,
	from model.Time,
	through model.Time,
	fn func(
		labels.Labels,
		model.Fingerprint,
		[]index.ChunkMeta,
	) (stop bool),
	matchers ...*labels.Matcher,
) error

type PowerOfTwoSharding

type PowerOfTwoSharding struct {
	MaxShards int
}

PowerOfTwoSharding is a slimmed down legacy sharding implementation designed for use as a fallback when the newer impls aren't found (i.e. during a system upgrade to support the new impl)

func (PowerOfTwoSharding) ShardsFor

func (p PowerOfTwoSharding) ShardsFor(bytes uint64, maxBytesPerShard uint64) []logproto.Shard

type SizedFP

type SizedFP struct {
	Fp    model.Fingerprint
	Stats stats.Stats
}

type SizedFPs

type SizedFPs []SizedFP

func (SizedFPs) Len

func (xs SizedFPs) Len() int

func (SizedFPs) Less

func (xs SizedFPs) Less(i, j int) bool

func (SizedFPs) ShardsFor

func (xs SizedFPs) ShardsFor(targetShardBytes uint64) (res []logproto.Shard)

func (SizedFPs) Swap

func (xs SizedFPs) Swap(i, j int)