chunk

Documentation

Index

• Constants
• Variables
• func RangeValues(it Iterator, in metric.Interval) ([]model.SamplePair, error)
• type Chunk
  • func New() Chunk
  • func NewForEncoding(encoding Encoding) (Chunk, error)
• type Desc
  • func NewDesc(c Chunk, firstTime model.Time) *Desc
  • func (d *Desc) Add(s model.SamplePair) ([]Chunk, error)
  • func (d *Desc) FirstTime() model.Time
  • func (d *Desc) IsEvicted() bool
  • func (d *Desc) LastTime() (model.Time, error)
  • func (d *Desc) MaybeEvict() bool
  • func (d *Desc) MaybePopulateLastTime() error
  • func (d *Desc) Pin(evictRequests chan<- EvictRequest)
  • func (d *Desc) RefCount() int
  • func (d *Desc) SetChunk(c Chunk)
  • func (d *Desc) Unpin(evictRequests chan<- EvictRequest)
• type Encoding
  • func (e *Encoding) Set(s string) error
  • func (e Encoding) String() string
• type EvictRequest
• type Iterator

Constants

const (

	// OpTypeLabel is the label name for chunk operation types.
	OpTypeLabel = "type"

	// CreateAndPin is the label value for create-and-pin chunk ops.
	CreateAndPin = "create" // A Desc creation with refCount=1.
	// PersistAndUnpin is the label value for persist chunk ops.
	PersistAndUnpin = "persist"
	// Pin is the label value for pin chunk ops (excludes pin on creation).
	Pin = "pin"
	// Unpin is the label value for unpin chunk ops (excludes the unpin on persisting).
	Unpin = "unpin"
	// Clone is the label value for clone chunk ops.
	Clone = "clone"
	// Transcode is the label value for transcode chunk ops.
	Transcode = "transcode"
	// Drop is the label value for drop chunk ops.
	Drop = "drop"

	// Evict is the label value for evict chunk desc ops.
	Evict = "evict"
	// Load is the label value for load chunk and chunk desc ops.
	Load = "load"
)

const ChunkLen = 1024

ChunkLen is the length of a chunk in bytes.

Variables

var (
	Ops = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: namespace,
			Subsystem: subsystem,
			Name:      "chunk_ops_total",
			Help:      "The total number of chunk operations by their type.",
		},
		[]string{OpTypeLabel},
	)
	DescOps = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: namespace,
			Subsystem: subsystem,
			Name:      "chunkdesc_ops_total",
			Help:      "The total number of chunk descriptor operations by their type.",
		},
		[]string{OpTypeLabel},
	)
	NumMemDescs = prometheus.NewGauge(prometheus.GaugeOpts{
		Namespace: namespace,
		Subsystem: subsystem,
		Name:      "memory_chunkdescs",
		Help:      "The current number of chunk descriptors in memory.",
	})
)

Usually, a separate file for instrumentation is frowned upon. Metrics should be close to where they are used. However, the metrics below are set all over the place, so we go for a separate instrumentation file in this case.

var (
	// NumMemChunks is the total number of chunks in memory. This is a
	// global counter, also used internally, so not implemented as
	// metrics. Collected in MemorySeriesStorage.Collect.
	// TODO(beorn7): As it is used internally, it is actually very bad style
	// to have it as a global variable.
	NumMemChunks int64

	// NumMemChunksDesc is the metric descriptor for the above.
	NumMemChunksDesc = prometheus.NewDesc(
		prometheus.BuildFQName(namespace, subsystem, "memory_chunks"),
		"The current number of chunks in memory, excluding cloned chunks (i.e. chunks without a descriptor).",
		nil, nil,
	)
)

var DefaultEncoding = DoubleDelta

DefaultEncoding can be changed via a flag.

Functions

func RangeValues

func RangeValues(it Iterator, in metric.Interval) ([]model.SamplePair, error)

RangeValues is a utility function that retrieves all values within the given range from an Iterator.

Types

type Chunk

type Chunk interface {
	// add adds a SamplePair to the chunks, performs any necessary
	// re-encoding, and adds any necessary overflow chunks. It returns the
	// new version of the original chunk, followed by overflow chunks, if
	// any. The first chunk returned might be the same as the original one
	// or a newly allocated version. In any case, take the returned chunk as
	// the relevant one and discard the original chunk.
	Add(sample model.SamplePair) ([]Chunk, error)
	Clone() Chunk
	FirstTime() model.Time
	NewIterator() Iterator
	Marshal(io.Writer) error
	MarshalToBuf([]byte) error
	Unmarshal(io.Reader) error
	UnmarshalFromBuf([]byte) error
	Encoding() Encoding
	Utilization() float64
}

Chunk is the interface for all chunks. Chunks are generally not goroutine-safe.

func New

func New() Chunk

New creates a new chunk according to the encoding set by the DefaultEncoding flag.

func NewForEncoding

func NewForEncoding(encoding Encoding) (Chunk, error)

NewForEncoding allows configuring what chunk type you want

type Desc

type Desc struct {
	sync.Mutex       // Protects pinning.
	C          Chunk // nil if chunk is evicted.

	ChunkFirstTime model.Time // Populated at creation. Immutable.
	ChunkLastTime  model.Time // Populated on closing of the chunk, model.Earliest if unset.

	// EvictListElement is nil if the chunk is not in the evict list.
	// EvictListElement is _not_ protected by the Desc mutex.
	// It must only be touched by the evict list handler in MemorySeriesStorage.
	EvictListElement *list.Element
	// contains filtered or unexported fields
}

Desc contains meta-data for a chunk. Pay special attention to the documented requirements for calling its methods concurrently (WRT pinning and locking). The doc comments spell out the requirements for each method, but here is an overview and general explanation:

Everything that changes the pinning of the underlying chunk or deals with its eviction is protected by a mutex. This affects the following methods: Pin, Unpin, RefCount, IsEvicted, MaybeEvict. These methods can be called at any time without further prerequisites.

Another group of methods acts on (or sets) the underlying chunk. These methods involve no locking. They may only be called if the caller has pinned the chunk (to guarantee the chunk is not evicted concurrently). Also, the caller must make sure nobody else will call these methods concurrently, either by holding the sole reference to the Desc (usually during loading or creation) or by locking the fingerprint of the series the Desc belongs to. The affected methods are: Add, MaybePopulateLastTime, SetChunk.

Finally, there are the special cases FirstTime and LastTime. LastTime requires to have locked the fingerprint of the series but the chunk does not need to be pinned. That's because the ChunkLastTime field in Desc gets populated upon completion of the chunk (when it is still pinned, and which happens while the series's fingerprint is locked). Once that has happened, calling LastTime does not require the chunk to be loaded anymore. Before that has happened, the chunk is pinned anyway. The ChunkFirstTime field in Desc is populated upon creation of a Desc, so it is alway safe to call FirstTime. The FirstTime method is arguably not needed and only there for consistency with LastTime.

func NewDesc

func NewDesc(c Chunk, firstTime model.Time) *Desc

NewDesc creates a new Desc pointing to the provided chunk. The provided chunk is assumed to be not persisted yet. Therefore, the refCount of the new Desc is 1 (preventing eviction prior to persisting).

func (*Desc) Add

func (d *Desc) Add(s model.SamplePair) ([]Chunk, error)

Add adds a sample pair to the underlying chunk. For safe concurrent access, The chunk must be pinned, and the caller must have locked the fingerprint of the series.

func (*Desc) FirstTime

func (d *Desc) FirstTime() model.Time

FirstTime returns the timestamp of the first sample in the chunk. This method can be called concurrently at any time. It only returns the immutable d.ChunkFirstTime without any locking. Arguably, this method is useless. However, it provides consistency with the LastTime method.

func (*Desc) IsEvicted

func (d *Desc) IsEvicted() bool

IsEvicted returns whether the chunk is evicted. For safe concurrent access, the caller must have locked the fingerprint of the series.

func (*Desc) LastTime

func (d *Desc) LastTime() (model.Time, error)

LastTime returns the timestamp of the last sample in the chunk. For safe concurrent access, this method requires the fingerprint of the time series to be locked.

func (*Desc) MaybeEvict

func (d *Desc) MaybeEvict() bool

MaybeEvict evicts the chunk if the refCount is 0. It returns whether the chunk is now evicted, which includes the case that the chunk was evicted even before this method was called. It can be called concurrently at any time.

func (*Desc) MaybePopulateLastTime

func (d *Desc) MaybePopulateLastTime() error

MaybePopulateLastTime populates the ChunkLastTime from the underlying chunk if it has not yet happened. Call this method directly after having added the last sample to a chunk or after closing a head chunk due to age. For safe concurrent access, the chunk must be pinned, and the caller must have locked the fingerprint of the series.

func (*Desc) Pin

func (d *Desc) Pin(evictRequests chan<- EvictRequest)

Pin increments the refCount by one. Upon increment from 0 to 1, this Desc is removed from the evict list. To enable the latter, the evictRequests channel has to be provided. This method can be called concurrently at any time.

func (*Desc) RefCount

func (d *Desc) RefCount() int

RefCount returns the number of pins. This method can be called concurrently at any time.

func (*Desc) SetChunk

func (d *Desc) SetChunk(c Chunk)

SetChunk sets the underlying chunk. The caller must have locked the fingerprint of the series and must have "pre-pinned" the chunk (i.e. first call Pin and then set the chunk).

func (*Desc) Unpin

func (d *Desc) Unpin(evictRequests chan<- EvictRequest)

Unpin decrements the refCount by one. Upon decrement from 1 to 0, this Desc is added to the evict list. To enable the latter, the evictRequests channel has to be provided. This method can be called concurrently at any time.

type Encoding

type Encoding byte

Encoding defines which encoding we are using, delta, doubledelta, or varbit

const (
	// Delta encoding
	Delta Encoding = iota
	// DoubleDelta encoding
	DoubleDelta
	// Varbit encoding
	Varbit
)

func (*Encoding) Set

func (e *Encoding) Set(s string) error

Set implements flag.Value.

func (Encoding) String

func (e Encoding) String() string

String implements flag.Value.

type EvictRequest

type EvictRequest struct {
	Desc  *Desc
	Evict bool
}

EvictRequest is a request to evict a chunk from memory.

type Iterator

type Iterator interface {
	// Gets the last timestamp in the chunk.
	LastTimestamp() (model.Time, error)
	// Whether a given timestamp is contained between first and last value
	// in the chunk.
	Contains(model.Time) (bool, error)
	// Scans the next value in the chunk. Directly after the iterator has
	// been created, the next value is the first value in the
	// chunk. Otherwise, it is the value following the last value scanned or
	// found (by one of the Find... methods). Returns false if either the
	// end of the chunk is reached or an error has occurred.
	Scan() bool
	// Finds the most recent value at or before the provided time. Returns
	// false if either the chunk contains no value at or before the provided
	// time, or an error has occurred.
	FindAtOrBefore(model.Time) bool
	// Finds the oldest value at or after the provided time. Returns false
	// if either the chunk contains no value at or after the provided time,
	// or an error has occurred.
	FindAtOrAfter(model.Time) bool
	// Returns the last value scanned (by the scan method) or found (by one
	// of the find... methods). It returns model.ZeroSamplePair before any of
	// those methods were called.
	Value() model.SamplePair
	// Returns the last error encountered. In general, an error signals data
	// corruption in the chunk and requires quarantining.
	Err() error
}

Iterator enables efficient access to the content of a chunk. It is generally not safe to use an Iterator concurrently with or after chunk mutation.