downsample

package
v0.37.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 25, 2024 License: Apache-2.0 Imports: 30 Imported by: 15

Documentation

Index

Constants

View Source 
const (
	ResLevel0 = int64(0)              // Raw data.
	ResLevel1 = int64(5 * 60 * 1000)  // 5 minutes in milliseconds.
	ResLevel2 = int64(60 * 60 * 1000) // 1 hour in milliseconds.
)

Standard downsampling resolution levels in Thanos.

View Source 
const (
	ResLevel1DownsampleRange = 40 * 60 * 60 * 1000      // 40 hours.
	ResLevel2DownsampleRange = 10 * 24 * 60 * 60 * 1000 // 10 days.
)

Downsampling ranges i.e. minimum block size after which we start to downsample blocks (in seconds).

View Source 
const ChunkEncAggr = chunkenc.Encoding(0xff)

ChunkEncAggr is the top level encoding byte for the AggrChunk. It picks the highest number possible to prevent future collisions with wrapped encodings.

Variables

View Source 
var ErrAggrNotExist = errors.New("aggregate does not exist")

ErrAggrNotExist is returned if a requested aggregation is not present in an AggrChunk.

Functions

func Downsample 

func Downsample(
	ctx context.Context,
	logger log.Logger,
	origMeta *metadata.Meta,
	b tsdb.BlockReader,
	dir string,
	resolution int64,
) (id ulid.ULID, err error)

Downsample downsamples the given block. It writes a new block into dir and returns its ID.

func DownsampleRaw added in v0.21.0 

func DownsampleRaw(data []sample, resolution int64) []chunks.Meta

DownsampleRaw create a series of aggregation chunks for the given sample data.

func NewPool 

func NewPool() chunkenc.Pool

TODO(bwplotka): Add reasonable limits to our sync pooling them to detect OOMs early.

func NewStreamedBlockWriter added in v0.3.0 

func NewStreamedBlockWriter(
	blockDir string,
	indexReader tsdb.IndexReader,
	logger log.Logger,
	originMeta metadata.Meta,
) (w *streamedBlockWriter, err error)

NewStreamedBlockWriter returns streamedBlockWriter instance, it's not concurrency safe. Caller is responsible to Close all io.Closers by calling the Close when downsampling is done. In case if error happens outside of the StreamedBlockWriter during the processing, index and meta files will be written anyway, so the caller is always responsible for removing block directory with a garbage on error. This approach simplifies StreamedBlockWriter interface, which is a best trade-off taking into account the error is an exception, not a general case.

func SamplesFromTSDBSamples added in v0.21.0 

func SamplesFromTSDBSamples(samples []chunks.Sample) []sample

SamplesFromTSDBSamples converts tsdbutil.Sample slice to samples.

Types

type AggrChunk 

type AggrChunk []byte

AggrChunk is a chunk that is composed of a set of aggregates for the same underlying data. Not all aggregates must be present.

func EncodeAggrChunk 

func EncodeAggrChunk(chks [5]chunkenc.Chunk) *AggrChunk

EncodeAggrChunk encodes a new aggregate chunk from the array of chunks for each aggregate. Each array entry corresponds to the respective AggrType number.

func (AggrChunk) Appender 

func (c AggrChunk) Appender() (chunkenc.Appender, error)

func (AggrChunk) Bytes 

func (c AggrChunk) Bytes() []byte

func (AggrChunk) Compact added in v0.13.0 

func (c AggrChunk) Compact()

func (AggrChunk) Encoding 

func (c AggrChunk) Encoding() chunkenc.Encoding

func (AggrChunk) Get 

func (c AggrChunk) Get(t AggrType) (chunkenc.Chunk, error)

Get returns the sub-chunk for the given aggregate type if it exists.

func (AggrChunk) Iterator 

func (c AggrChunk) Iterator(_ chunkenc.Iterator) chunkenc.Iterator

func (AggrChunk) NumSamples 

func (c AggrChunk) NumSamples() int

func (*AggrChunk) Reset added in v0.36.0 

func (c *AggrChunk) Reset(stream []byte)

type AggrType 

type AggrType uint8

AggrType represents an aggregation type. 

const (
	AggrCount AggrType = iota
	AggrSum
	AggrMin
	AggrMax
	AggrCounter
)

Valid aggregations.

func (AggrType) String 

func (t AggrType) String() string

type ApplyCounterResetsSeriesIterator added in v0.13.0 

type ApplyCounterResetsSeriesIterator struct {
	// contains filtered or unexported fields
}

ApplyCounterResetsSeriesIterator generates monotonically increasing values by iterating over an ordered sequence of chunks, which should be raw or aggregated chunks of counter values. The generated samples can be used by PromQL functions like 'rate' that calculate differences between counter values. Stale Markers are removed as well.

Counter aggregation chunks must have the first and last values from their original raw series: the first raw value should be the first value encoded in the chunk, and the last raw value is encoded by the duplication of the previous sample's timestamp. As iteration occurs between chunks, the comparison between the last raw value of the earlier chunk and the first raw value of the later chunk ensures that counter resets between chunks are recognized and that the correct value delta is calculated.

It handles overlapped chunks (removes overlaps). NOTE: It is important to deduplicate with care ensuring that you don't hit issue https://github.com/thanos-io/thanos/issues/2401#issuecomment-621958839. NOTE(bwplotka): This hides resets from PromQL engine. This means it will not work for PromQL resets function.

func NewApplyCounterResetsIterator added in v0.13.0 

func NewApplyCounterResetsIterator(chks ...chunkenc.Iterator) *ApplyCounterResetsSeriesIterator

func (*ApplyCounterResetsSeriesIterator) At added in v0.13.0 

func (it *ApplyCounterResetsSeriesIterator) At() (t int64, v float64)

func (*ApplyCounterResetsSeriesIterator) AtFloatHistogram added in v0.30.0 

func (it *ApplyCounterResetsSeriesIterator) AtFloatHistogram(fh *histogram.FloatHistogram) (int64, *histogram.FloatHistogram)

func (*ApplyCounterResetsSeriesIterator) AtHistogram added in v0.30.0 

func (it *ApplyCounterResetsSeriesIterator) AtHistogram(h *histogram.Histogram) (int64, *histogram.Histogram)

func (*ApplyCounterResetsSeriesIterator) AtT added in v0.30.0 

func (it *ApplyCounterResetsSeriesIterator) AtT() int64

func (*ApplyCounterResetsSeriesIterator) Err added in v0.13.0 

func (it *ApplyCounterResetsSeriesIterator) Err() error

func (*ApplyCounterResetsSeriesIterator) Next added in v0.13.0 

func (it *ApplyCounterResetsSeriesIterator) Next() chunkenc.ValueType

TODO(rabenhorst): Native histogram support needs to be added, float type is hardcoded.

func (*ApplyCounterResetsSeriesIterator) Seek added in v0.13.0 

func (it *ApplyCounterResetsSeriesIterator) Seek(x int64) chunkenc.ValueType

type AverageChunkIterator 

type AverageChunkIterator struct {
	// contains filtered or unexported fields
}

AverageChunkIterator emits an artificial series of average samples based in aggregate chunks with sum and count aggregates.

func NewAverageChunkIterator 

func NewAverageChunkIterator(cnt, sum chunkenc.Iterator) *AverageChunkIterator

func (*AverageChunkIterator) At 

func (it *AverageChunkIterator) At() (int64, float64)

func (*AverageChunkIterator) AtFloatHistogram added in v0.30.0 

func (it *AverageChunkIterator) AtFloatHistogram(*histogram.FloatHistogram) (int64, *histogram.FloatHistogram)

func (*AverageChunkIterator) AtHistogram added in v0.30.0 

func (it *AverageChunkIterator) AtHistogram(*histogram.Histogram) (int64, *histogram.Histogram)

TODO(rabenhorst): Needs to be implemented for native histogram support.

func (*AverageChunkIterator) AtT added in v0.30.0 

func (it *AverageChunkIterator) AtT() int64

func (*AverageChunkIterator) Err 

func (it *AverageChunkIterator) Err() error

func (*AverageChunkIterator) Next 

func (it *AverageChunkIterator) Next() chunkenc.ValueType

TODO(rabenhorst): Native histogram support needs to be added, float type is hardcoded.

func (*AverageChunkIterator) Seek added in v0.13.0 

func (it *AverageChunkIterator) Seek(t int64) chunkenc.ValueType

type GatherNoDownsampleMarkFilter added in v0.30.0 

type GatherNoDownsampleMarkFilter struct {
	// contains filtered or unexported fields
}

GatherNoDownsampleMarkFilter is a block.Fetcher filter that passes all metas. While doing it, it gathers all no-downsample-mark.json markers.

func NewGatherNoDownsampleMarkFilter added in v0.30.0 

func NewGatherNoDownsampleMarkFilter(logger log.Logger, bkt objstore.InstrumentedBucketReader, concurrency int) *GatherNoDownsampleMarkFilter

NewGatherNoDownsampleMarkFilter creates GatherNoDownsampleMarkFilter.

func (*GatherNoDownsampleMarkFilter) Filter added in v0.30.0 

func (f *GatherNoDownsampleMarkFilter) Filter(ctx context.Context, metas map[ulid.ULID]*metadata.Meta, synced block.GaugeVec, modified block.GaugeVec) error

TODO (@rohitkochhar): reduce code duplication here by combining this code with that of GatherNoCompactionMarkFilter Filter passes all metas, while gathering no downsample markers.

func (*GatherNoDownsampleMarkFilter) NoDownsampleMarkedBlocks added in v0.30.0 

func (f *GatherNoDownsampleMarkFilter) NoDownsampleMarkedBlocks() map[ulid.ULID]*metadata.NoDownsampleMark

NoDownsampleMarkedBlocks returns block ids that were marked for no downsample.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.