encoding

Documentation

Overview

This file was taken from Prometheus (https://github.com/prometheus/prometheus). The original license header is included below:

Copyright 2016 The Prometheus Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. nolint //Since this was copied from Prometheus leave it as is

This file was taken from Prometheus (https://github.com/prometheus/prometheus). The original license header is included below:

Copyright 2016 The Prometheus Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. nolint //Since this was copied from Prometheus leave it as is

Index

• Constants
• Variables
• func MustRegisterEncoding(enc Encoding, name string, f func() Chunk)
• func RangeValues(it Iterator, in metric.Interval) ([]model.SamplePair, error)
• type Batch
• type Chunk
  • func New() Chunk
  • func NewForEncoding(encoding Encoding) (Chunk, error)
• type Config
  • func (Config) RegisterFlags(f *flag.FlagSet)
  • func (Config) Validate() error
• type Encoding
  • func (e *Encoding) Set(s string) error
  • func (e Encoding) String() string
• type Iterator

Constants

const (
	// ChunkLen is the length of a chunk in bytes.
	ChunkLen = 1024

	ErrSliceNoDataInRange = errs.Error("chunk has no data for given range to slice")
	ErrSliceChunkOverflow = errs.Error("slicing should not overflow a chunk")
)

const (

	// OpTypeLabel is the label name for chunk operation types.
	OpTypeLabel = "type"

	// CreateAndPin is the label value for create-and-pin chunk ops.
	CreateAndPin = "create" // A Desc creation with refCount=1.
	// PersistAndUnpin is the label value for persist chunk ops.
	PersistAndUnpin = "persist"
	// Pin is the label value for pin chunk ops (excludes pin on creation).
	Pin = "pin"
	// Unpin is the label value for unpin chunk ops (excludes the unpin on persisting).
	Unpin = "unpin"
	// Transcode is the label value for transcode chunk ops.
	Transcode = "transcode"
	// Drop is the label value for drop chunk ops.
	Drop = "drop"

	// Evict is the label value for evict chunk desc ops.
	Evict = "evict"
	// Load is the label value for load chunk and chunk desc ops.
	Load = "load"
)

const BatchSize = 12

BatchSize is samples per batch; this was choose by benchmarking all sizes from 1 to 128.

Variables

var (
	Ops = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: namespace,
			Subsystem: subsystem,
			Name:      "chunk_ops_total",
			Help:      "The total number of chunk operations by their type.",
		},
		[]string{OpTypeLabel},
	)
	DescOps = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: namespace,
			Subsystem: subsystem,
			Name:      "chunkdesc_ops_total",
			Help:      "The total number of chunk descriptor operations by their type.",
		},
		[]string{OpTypeLabel},
	)
	NumMemDescs = prometheus.NewGauge(prometheus.GaugeOpts{
		Namespace: namespace,
		Subsystem: subsystem,
		Name:      "memory_chunkdescs",
		Help:      "The current number of chunk descriptors in memory.",
	})
)

Usually, a separate file for instrumentation is frowned upon. Metrics should be close to where they are used. However, the metrics below are set all over the place, so we go for a separate instrumentation file in this case.

var (
	// DefaultEncoding exported for use in unit tests elsewhere
	DefaultEncoding = Bigchunk
)

var NumMemChunks int64

NumMemChunks is the total number of chunks in memory. This is a global counter, also used internally, so not implemented as metrics. Collected in MemorySeriesStorage. TODO(beorn7): Having this as an exported global variable is really bad.

Functions

func MustRegisterEncoding

func MustRegisterEncoding(enc Encoding, name string, f func() Chunk)

MustRegisterEncoding add a new chunk encoding. There is no locking, so this must be called in init().

func RangeValues

func RangeValues(it Iterator, in metric.Interval) ([]model.SamplePair, error)

RangeValues is a utility function that retrieves all values within the given range from an Iterator.

Types

type Batch

type Batch struct {
	Timestamps [BatchSize]int64
	Values     [BatchSize]float64
	Index      int
	Length     int
}

Batch is a sorted set of (timestamp, value) pairs. They are intended to be small, and passed by value.

type Chunk

type Chunk interface {
	// Add adds a SamplePair to the chunks, performs any necessary
	// re-encoding, and creates any necessary overflow chunk.
	// The returned Chunk is the overflow chunk if it was created.
	// The returned Chunk is nil if the sample got appended to the same chunk.
	Add(sample model.SamplePair) (Chunk, error)
	// NewIterator returns an iterator for the chunks.
	// The iterator passed as argument is for re-use. Depending on implementation,
	// the iterator can be re-used or a new iterator can be allocated.
	NewIterator(Iterator) Iterator
	Marshal(io.Writer) error
	UnmarshalFromBuf([]byte) error
	Encoding() Encoding
	Utilization() float64

	// Slice returns a smaller chunk that includes all samples between start and end
	// (inclusive).  Its may over estimate. On some encodings it is a noop.
	Slice(start, end model.Time) Chunk

	// Rebound returns a smaller chunk that includes all samples between start and end (inclusive).
	// We do not want to change existing Slice implementations because
	// it is built specifically for query optimization and is a noop for some of the encodings.
	Rebound(start, end model.Time) (Chunk, error)

	// Len returns the number of samples in the chunk.  Implementations may be
	// expensive.
	Len() int

	// Size returns the approximate length of the chunk in bytes.
	Size() int
}

Chunk is the interface for all chunks. Chunks are generally not goroutine-safe.

func New

func New() Chunk

New creates a new chunk according to the encoding set by the DefaultEncoding flag.

func NewForEncoding

func NewForEncoding(encoding Encoding) (Chunk, error)

NewForEncoding allows configuring what chunk type you want

type Config

type Config struct{}

Config configures the behaviour of chunk encoding

func (Config) RegisterFlags

func (Config) RegisterFlags(f *flag.FlagSet)

RegisterFlags registers configuration settings.

func (Config) Validate

func (Config) Validate() error

Validate errors out if the encoding is set to Delta.

type Encoding

type Encoding byte

Encoding defines which encoding we are using, delta, doubledelta, or varbit

const (
	// Delta encoding is no longer supported and will be automatically changed to DoubleDelta.
	// It still exists here to not change the `ingester.chunk-encoding` flag values.
	Delta Encoding = iota
	// DoubleDelta encoding
	DoubleDelta
	// Varbit encoding
	Varbit
	// Bigchunk encoding
	Bigchunk
	// PrometheusXorChunk is a wrapper around Prometheus XOR-encoded chunk.
	PrometheusXorChunk
)

func (*Encoding) Set

func (e *Encoding) Set(s string) error

Set implements flag.Value.

func (Encoding) String

func (e Encoding) String() string

String implements flag.Value.

type Iterator

type Iterator interface {
	// Scans the next value in the chunk. Directly after the iterator has
	// been created, the next value is the first value in the
	// chunk. Otherwise, it is the value following the last value scanned or
	// found (by one of the Find... methods). Returns false if either the
	// end of the chunk is reached or an error has occurred.
	Scan() bool
	// Finds the oldest value at or after the provided time. Returns false
	// if either the chunk contains no value at or after the provided time,
	// or an error has occurred.
	FindAtOrAfter(model.Time) bool
	// Returns the last value scanned (by the scan method) or found (by one
	// of the find... methods). It returns model.ZeroSamplePair before any of
	// those methods were called.
	Value() model.SamplePair
	// Returns a batch of the provisded size; NB not idempotent!  Should only be called
	// once per Scan.
	Batch(size int) Batch
	// Returns the last error encountered. In general, an error signals data
	// corruption in the chunk and requires quarantining.
	Err() error
}

Iterator enables efficient access to the content of a chunk. It is generally not safe to use an Iterator concurrently with or after chunk mutation.