chunkenc

package
v0.54.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2024 License: Apache-2.0 Imports: 8 Imported by: 152

Documentation

Index

Constants

View Source 
const (
	// MaxBytesPerXORChunk is the maximum size an XOR chunk can be.
	MaxBytesPerXORChunk = 1024
	// TargetBytesPerHistogramChunk sets a size target for each histogram chunk.
	TargetBytesPerHistogramChunk = 1024
	// MinSamplesPerHistogramChunk sets a minimum sample count for histogram chunks. This is desirable because a single
	// histogram sample can be larger than TargetBytesPerHistogramChunk but we want to avoid too-small sample count
	// chunks so we can achieve some measure of compression advantage even while dealing with really large histograms.
	// Note that this minimum sample count is not enforced across chunk range boundaries (for example, if the chunk
	// range is 100 and the first sample in the chunk range is 99, the next sample will be included in a new chunk
	// resulting in the old chunk containing only a single sample).
	MinSamplesPerHistogramChunk = 10
)
View Source 
const CounterResetHeaderMask byte = 0b11000000

CounterResetHeaderMask is the mask to get the counter reset header bits.

Variables

This section is empty.

Functions

func IsValidEncoding added in v0.39.0 

func IsValidEncoding(e Encoding) bool

IsValidEncoding returns true for supported encodings.

Types

type Appender 

type Appender interface {
	Append(int64, float64)

	// AppendHistogram and AppendFloatHistogram append a histogram sample to a histogram or float histogram chunk.
	// Appending a histogram may require creating a completely new chunk or recoding (changing) the current chunk.
	// The Appender prev is used to determine if there is a counter reset between the previous Appender and the current Appender.
	// The Appender prev is optional and only taken into account when the first sample is being appended.
	// The bool appendOnly governs what happens when a sample cannot be appended to the current chunk. If appendOnly is true, then
	// in such case an error is returned without modifying the chunk. If appendOnly is false, then a new chunk is created or the
	// current chunk is recoded to accommodate the sample.
	// The returned Chunk c is nil if sample could be appended to the current Chunk, otherwise c is the new Chunk.
	// The returned bool isRecoded can be used to distinguish between the new Chunk c being a completely new Chunk
	// or the current Chunk recoded to a new Chunk.
	// The Appender app that can be used for the next append is always returned.
	AppendHistogram(prev *HistogramAppender, t int64, h *histogram.Histogram, appendOnly bool) (c Chunk, isRecoded bool, app Appender, err error)
	AppendFloatHistogram(prev *FloatHistogramAppender, t int64, h *histogram.FloatHistogram, appendOnly bool) (c Chunk, isRecoded bool, app Appender, err error)
}

Appender adds sample pairs to a chunk.

type Chunk 

type Chunk interface {
	Iterable

	// Bytes returns the underlying byte slice of the chunk.
	Bytes() []byte

	// Encoding returns the encoding type of the chunk.
	Encoding() Encoding

	// Appender returns an appender to append samples to the chunk.
	Appender() (Appender, error)

	// NumSamples returns the number of samples in the chunk.
	NumSamples() int

	// Compact is called whenever a chunk is expected to be complete (no more
	// samples appended) and the underlying implementation can eventually
	// optimize the chunk.
	// There's no strong guarantee that no samples will be appended once
	// Compact() is called. Implementing this function is optional.
	Compact()

	// Reset resets the chunk given stream.
	Reset(stream []byte)
}

Chunk holds a sequence of sample pairs that can be iterated over and appended to.

func FromData 

func FromData(e Encoding, d []byte) (Chunk, error)

FromData returns a chunk from a byte slice of chunk data. This is there so that users of the library can easily create chunks from bytes.

func NewEmptyChunk added in v0.40.0 

func NewEmptyChunk(e Encoding) (Chunk, error)

NewEmptyChunk returns an empty chunk for the given encoding.

type CounterResetHeader added in v0.40.0 

type CounterResetHeader byte

CounterResetHeader defines the first 2 bits of the chunk header. 

const (
	// CounterReset means there was definitely a counter reset that resulted in this chunk.
	CounterReset CounterResetHeader = 0b10000000
	// NotCounterReset means there was definitely no counter reset when cutting this chunk.
	NotCounterReset CounterResetHeader = 0b01000000
	// GaugeType means this chunk contains a gauge histogram, where counter resets do not happen.
	GaugeType CounterResetHeader = 0b11000000
	// UnknownCounterReset means we cannot say if this chunk was created due to a counter reset or not.
	// An explicit counter reset detection needs to happen during query time.
	UnknownCounterReset CounterResetHeader = 0b00000000
)

func CounterResetHintToHeader added in v0.47.0 

func CounterResetHintToHeader(hint histogram.CounterResetHint) CounterResetHeader

type Encoding 

type Encoding uint8

Encoding is the identifier for a chunk encoding. 

const (
	EncNone Encoding = iota
	EncXOR
	EncHistogram
	EncFloatHistogram
)

The different available chunk encodings.

func (Encoding) String 

func (e Encoding) String() string

type FloatHistogramAppender added in v0.42.0 

type FloatHistogramAppender struct {
	// contains filtered or unexported fields
}

FloatHistogramAppender is an Appender implementation for float histograms.

func (*FloatHistogramAppender) Append added in v0.42.0 

func (a *FloatHistogramAppender) Append(int64, float64)

Append implements Appender. This implementation panics because normal float samples must never be appended to a histogram chunk.

func (*FloatHistogramAppender) AppendFloatHistogram added in v0.42.0 

func (a *FloatHistogramAppender) AppendFloatHistogram(prev *FloatHistogramAppender, t int64, h *histogram.FloatHistogram, appendOnly bool) (Chunk, bool, Appender, error)

func (*FloatHistogramAppender) AppendHistogram added in v0.42.0 

func (a *FloatHistogramAppender) AppendHistogram(*HistogramAppender, int64, *histogram.Histogram, bool) (Chunk, bool, Appender, error)

func (*FloatHistogramAppender) GetCounterResetHeader added in v0.42.0 

func (a *FloatHistogramAppender) GetCounterResetHeader() CounterResetHeader

func (*FloatHistogramAppender) NumSamples added in v0.42.0 

func (a *FloatHistogramAppender) NumSamples() int

type FloatHistogramChunk added in v0.42.0 

type FloatHistogramChunk struct {
	// contains filtered or unexported fields
}

FloatHistogramChunk holds encoded sample data for a sparse, high-resolution float histogram.

Each sample has multiple "fields", stored in the following way (raw = store number directly, delta = store delta to the previous number, dod = store delta of the delta to the previous number, xor = what we do for regular sample values): 

field →    ts    count zeroCount sum []posbuckets []negbuckets
sample 1   raw   raw   raw       raw []raw        []raw
sample 2   delta xor   xor       xor []xor        []xor
sample >2  dod   xor   xor       xor []xor        []xor

func NewFloatHistogramChunk added in v0.42.0 

func NewFloatHistogramChunk() *FloatHistogramChunk

NewFloatHistogramChunk returns a new chunk with float histogram encoding.

func (*FloatHistogramChunk) Appender added in v0.42.0 

func (c *FloatHistogramChunk) Appender() (Appender, error)

Appender implements the Chunk interface.

func (*FloatHistogramChunk) Bytes added in v0.42.0 

func (c *FloatHistogramChunk) Bytes() []byte

Bytes returns the underlying byte slice of the chunk.

func (*FloatHistogramChunk) Compact added in v0.42.0 

func (c *FloatHistogramChunk) Compact()

Compact implements the Chunk interface.

func (*FloatHistogramChunk) Encoding added in v0.42.0 

func (c *FloatHistogramChunk) Encoding() Encoding

Encoding returns the encoding type.

func (*FloatHistogramChunk) GetCounterResetHeader added in v0.42.0 

func (c *FloatHistogramChunk) GetCounterResetHeader() CounterResetHeader

GetCounterResetHeader returns the info about the first 2 bits of the chunk header.

func (*FloatHistogramChunk) Iterator added in v0.42.0 

func (c *FloatHistogramChunk) Iterator(it Iterator) Iterator

Iterator implements the Chunk interface.

func (*FloatHistogramChunk) Layout added in v0.42.0 

func (c *FloatHistogramChunk) Layout() (
	schema int32, zeroThreshold float64,
	negativeSpans, positiveSpans []histogram.Span,
	customValues []float64,
	err error,
)

Layout returns the histogram layout. Only call this on chunks that have at least one sample.

func (*FloatHistogramChunk) NumSamples added in v0.42.0 

func (c *FloatHistogramChunk) NumSamples() int

NumSamples returns the number of samples in the chunk.

func (*FloatHistogramChunk) Reset added in v0.53.0 

func (c *FloatHistogramChunk) Reset(stream []byte)

type HistogramAppender added in v0.40.0 

type HistogramAppender struct {
	// contains filtered or unexported fields
}

HistogramAppender is an Appender implementation for sparse histograms.

func (*HistogramAppender) Append added in v0.40.0 

func (a *HistogramAppender) Append(int64, float64)

Append implements Appender. This implementation panics because normal float samples must never be appended to a histogram chunk.

func (*HistogramAppender) AppendFloatHistogram added in v0.42.0 

func (a *HistogramAppender) AppendFloatHistogram(*FloatHistogramAppender, int64, *histogram.FloatHistogram, bool) (Chunk, bool, Appender, error)

func (*HistogramAppender) AppendHistogram added in v0.40.0 

func (a *HistogramAppender) AppendHistogram(prev *HistogramAppender, t int64, h *histogram.Histogram, appendOnly bool) (Chunk, bool, Appender, error)

func (*HistogramAppender) GetCounterResetHeader added in v0.42.0 

func (a *HistogramAppender) GetCounterResetHeader() CounterResetHeader

func (*HistogramAppender) NumSamples added in v0.42.0 

func (a *HistogramAppender) NumSamples() int

type HistogramChunk added in v0.40.0 

type HistogramChunk struct {
	// contains filtered or unexported fields
}

HistogramChunk holds encoded sample data for a sparse, high-resolution histogram.

Each sample has multiple "fields", stored in the following way (raw = store number directly, delta = store delta to the previous number, dod = store delta of the delta to the previous number, xor = what we do for regular sample values): 

field →    ts    count zeroCount sum []posbuckets []negbuckets
sample 1   raw   raw   raw       raw []raw        []raw
sample 2   delta delta delta     xor []delta      []delta
sample >2  dod   dod   dod       xor []dod        []dod

func NewHistogramChunk added in v0.40.0 

func NewHistogramChunk() *HistogramChunk

NewHistogramChunk returns a new chunk with histogram encoding of the given size.

func (*HistogramChunk) Appender added in v0.40.0 

func (c *HistogramChunk) Appender() (Appender, error)

Appender implements the Chunk interface.

func (*HistogramChunk) Bytes added in v0.40.0 

func (c *HistogramChunk) Bytes() []byte

Bytes returns the underlying byte slice of the chunk.

func (*HistogramChunk) Compact added in v0.40.0 

func (c *HistogramChunk) Compact()

Compact implements the Chunk interface.

func (*HistogramChunk) Encoding added in v0.40.0 

func (c *HistogramChunk) Encoding() Encoding

Encoding returns the encoding type.

func (*HistogramChunk) GetCounterResetHeader added in v0.40.0 

func (c *HistogramChunk) GetCounterResetHeader() CounterResetHeader

GetCounterResetHeader returns the info about the first 2 bits of the chunk header.

func (*HistogramChunk) Iterator added in v0.40.0 

func (c *HistogramChunk) Iterator(it Iterator) Iterator

Iterator implements the Chunk interface.

func (*HistogramChunk) Layout added in v0.40.0 

func (c *HistogramChunk) Layout() (
	schema int32, zeroThreshold float64,
	negativeSpans, positiveSpans []histogram.Span,
	customValues []float64,
	err error,
)

Layout returns the histogram layout. Only call this on chunks that have at least one sample.

func (*HistogramChunk) NumSamples added in v0.40.0 

func (c *HistogramChunk) NumSamples() int

NumSamples returns the number of samples in the chunk.

func (*HistogramChunk) Reset added in v0.53.0 

func (c *HistogramChunk) Reset(stream []byte)

type Insert added in v0.42.0 

type Insert struct {
	// contains filtered or unexported fields
}

An Insert describes how many new buckets have to be inserted before processing the pos'th bucket from the original slice.

type Iterable added in v0.49.0 

type Iterable interface {
	// The iterator passed as argument is for re-use.
	// Depending on implementation, the iterator can
	// be re-used or a new iterator can be allocated.
	Iterator(Iterator) Iterator
}

type Iterator 

type Iterator interface {
	// Next advances the iterator by one and returns the type of the value
	// at the new position (or ValNone if the iterator is exhausted).
	Next() ValueType
	// Seek advances the iterator forward to the first sample with a
	// timestamp equal or greater than t. If the current sample found by a
	// previous `Next` or `Seek` operation already has this property, Seek
	// has no effect. If a sample has been found, Seek returns the type of
	// its value. Otherwise, it returns ValNone, after which the iterator is
	// exhausted.
	Seek(t int64) ValueType
	// At returns the current timestamp/value pair if the value is a float.
	// Before the iterator has advanced, the behaviour is unspecified.
	At() (int64, float64)
	// AtHistogram returns the current timestamp/value pair if the value is a
	// histogram with integer counts. Before the iterator has advanced, the behaviour
	// is unspecified.
	// The method accepts an optional Histogram object which will be
	// reused when not nil. Otherwise, a new Histogram object will be allocated.
	AtHistogram(*histogram.Histogram) (int64, *histogram.Histogram)
	// AtFloatHistogram returns the current timestamp/value pair if the
	// value is a histogram with floating-point counts. It also works if the
	// value is a histogram with integer counts, in which case a
	// FloatHistogram copy of the histogram is returned. Before the iterator
	// has advanced, the behaviour is unspecified.
	// The method accepts an optional FloatHistogram object which will be
	// reused when not nil. Otherwise, a new FloatHistogram object will be allocated.
	AtFloatHistogram(*histogram.FloatHistogram) (int64, *histogram.FloatHistogram)
	// AtT returns the current timestamp.
	// Before the iterator has advanced, the behaviour is unspecified.
	AtT() int64
	// Err returns the current error. It should be used only after the
	// iterator is exhausted, i.e. `Next` or `Seek` have returned ValNone.
	Err() error
}

Iterator is a simple iterator that can only get the next value. Iterator iterates over the samples of a time series, in timestamp-increasing order.

func MockSeriesIterator 

func MockSeriesIterator(timestamps []int64, values []float64) Iterator

MockSeriesIterator returns an iterator for a mock series with custom timeStamps and values.

func NewNopIterator 

func NewNopIterator() Iterator

NewNopIterator returns a new chunk iterator that does not hold any data.

type Pool 

type Pool interface {
	Put(Chunk) error
	Get(e Encoding, b []byte) (Chunk, error)
}

Pool is used to create and reuse chunk references to avoid allocations.

func NewPool 

func NewPool() Pool

NewPool returns a new pool.

type ValueType added in v0.40.0 

type ValueType uint8

ValueType defines the type of a value an Iterator points to. 

const (
	ValNone           ValueType = iota // No value at the current position.
	ValFloat                           // A simple float, retrieved with At.
	ValHistogram                       // A histogram, retrieve with AtHistogram, but AtFloatHistogram works, too.
	ValFloatHistogram                  // A floating-point histogram, retrieve with AtFloatHistogram.
)

Possible values for ValueType.

func (ValueType) ChunkEncoding added in v0.40.0 

func (v ValueType) ChunkEncoding() Encoding

func (ValueType) NewChunk added in v0.49.0 

func (v ValueType) NewChunk() (Chunk, error)

func (ValueType) String added in v0.40.0 

func (v ValueType) String() string

type XORChunk 

type XORChunk struct {
	// contains filtered or unexported fields
}

XORChunk holds XOR encoded sample data.

func NewXORChunk 

func NewXORChunk() *XORChunk

NewXORChunk returns a new chunk with XOR encoding.

func (*XORChunk) Appender 

func (c *XORChunk) Appender() (Appender, error)

Appender implements the Chunk interface. It is not valid to call Appender() multiple times concurrently or to use multiple Appenders on the same chunk.

func (*XORChunk) Bytes 

func (c *XORChunk) Bytes() []byte

Bytes returns the underlying byte slice of the chunk.

func (*XORChunk) Compact 

func (c *XORChunk) Compact()

Compact implements the Chunk interface.

func (*XORChunk) Encoding 

func (c *XORChunk) Encoding() Encoding

Encoding returns the encoding type.

func (*XORChunk) Iterator 

func (c *XORChunk) Iterator(it Iterator) Iterator

Iterator implements the Chunk interface. Iterator() must not be called concurrently with any modifications to the chunk, but after it returns you can use an Iterator concurrently with an Appender or other Iterators.

func (*XORChunk) NumSamples 

func (c *XORChunk) NumSamples() int

NumSamples returns the number of samples in the chunk.

func (*XORChunk) Reset added in v0.53.0 

func (c *XORChunk) Reset(stream []byte)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.