storage

package
v1.52.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 13, 2021 License: Apache-2.0 Imports: 37 Imported by: 10

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// MetricsWithDroppedLabels is the number of metrics with at least a single dropped label
	MetricsWithDroppedLabels uint64

	// TooLongLabelNames is the number of too long label names
	TooLongLabelNames uint64

	// TooLongLabelValues is the number of too long label values
	TooLongLabelValues uint64
)
View Source 
var ErrDeadlineExceeded = fmt.Errorf("deadline exceeded")

ErrDeadlineExceeded is returned when the request times out.

Functions

func DeduplicateSamples added in v1.33.0 

func DeduplicateSamples(srcTimestamps []int64, srcValues []float64) ([]int64, []float64)

DeduplicateSamples removes samples from src* if they are closer to each other than minScrapeInterval.

func MarshalMetricNameRaw 

func MarshalMetricNameRaw(dst []byte, labels []prompb.Label) []byte

MarshalMetricNameRaw marshals labels to dst and returns the result.

The result must be unmarshaled with MetricName.unmarshalRaw

func PutMetricName 

func PutMetricName(mn *MetricName)

PutMetricName returns mn to the pool.

func RegexpCacheMisses 

func RegexpCacheMisses() uint64

RegexpCacheMisses returns the number of cache misses for regexp cache.

func RegexpCacheRequests 

func RegexpCacheRequests() uint64

RegexpCacheRequests returns the number of requests to regexp cache.

func RegexpCacheSize 

func RegexpCacheSize() int

RegexpCacheSize returns the number of cached regexps for tag filters.

func SetBigMergeWorkersCount added in v1.28.1 

func SetBigMergeWorkersCount(n int)

SetBigMergeWorkersCount sets the maximum number of concurrent mergers for big blocks.

The function must be called before opening or creating any storage.

func SetFinalMergeDelay added in v1.44.0 

func SetFinalMergeDelay(delay time.Duration)

SetFinalMergeDelay sets the delay before doing final merge for partitions without newly ingested data.

This function may be called only before Storage initialization.

func SetMaxLabelsPerTimeseries added in v1.27.0 

func SetMaxLabelsPerTimeseries(maxLabels int)

SetMaxLabelsPerTimeseries sets the limit on the number of labels per each time series.

Superfluous labels are dropped.

func SetMinScrapeIntervalForDeduplication added in v1.33.1 

func SetMinScrapeIntervalForDeduplication(interval time.Duration)

SetMinScrapeIntervalForDeduplication sets the minimum interval for data points during de-duplication.

De-duplication is disabled if interval is 0.

This function must be called before initializing the storage.

func SetSmallMergeWorkersCount added in v1.28.1 

func SetSmallMergeWorkersCount(n int)

SetSmallMergeWorkersCount sets the maximum number of concurrent mergers for small blocks.

The function must be called before opening or creating any storage.

Types

type Block 

type Block struct {
	// contains filtered or unexported fields
}

Block represents a block of time series values for a single TSID.

func (*Block) AppendRowsWithTimeRangeFilter added in v1.42.0 

func (b *Block) AppendRowsWithTimeRangeFilter(dstTimestamps []int64, dstValues []float64, tr TimeRange) ([]int64, []float64)

AppendRowsWithTimeRangeFilter filters samples from b according to tr and appends them to dst*.

It is expected that UnmarshalData has been already called on b.

func (*Block) CopyFrom 

func (b *Block) CopyFrom(src *Block)

CopyFrom copies src to b.

func (*Block) Init 

func (b *Block) Init(tsid *TSID, timestamps, values []int64, scale int16, precisionBits uint8)

Init initializes b with the given tsid, timestamps, values and scale.

func (*Block) MarshalData 

func (b *Block) MarshalData(timestampsBlockOffset, valuesBlockOffset uint64) ([]byte, []byte, []byte)

MarshalData marshals the block into binary representation.

func (*Block) MarshalPortable added in v1.42.0 

func (b *Block) MarshalPortable(dst []byte) []byte

MarshalPortable marshals b to dst, so it could be portably migrated to other VictoriaMetrics instance.

The marshaled value must be unmarshaled with UnmarshalPortable function.

func (*Block) Reset 

func (b *Block) Reset()

Reset resets b.

func (*Block) RowsCount 

func (b *Block) RowsCount() int

RowsCount returns the number of rows in the block.

func (*Block) UnmarshalData 

func (b *Block) UnmarshalData() error

UnmarshalData unmarshals block data.

func (*Block) UnmarshalPortable added in v1.42.0 

func (b *Block) UnmarshalPortable(src []byte) ([]byte, error)

UnmarshalPortable unmarshals block from src to b and returns the remaining tail.

It is assumed that the block has been marshaled with MarshalPortable.

type BlockRef added in v1.35.0 

type BlockRef struct {
	// contains filtered or unexported fields
}

BlockRef references a Block.

BlockRef is valid only until the corresponding Search is valid, i.e. it becomes invalid after Search.MustClose is called.

func (*BlockRef) Init added in v1.46.0 

func (br *BlockRef) Init(pr PartRef, data []byte) error

Init initializes br from pr and data

func (*BlockRef) Marshal added in v1.46.0 

func (br *BlockRef) Marshal(dst []byte) []byte

Marshal marshals br to dst.

func (*BlockRef) MustReadBlock added in v1.35.0 

func (br *BlockRef) MustReadBlock(dst *Block, fetchData bool)

MustReadBlock reads block from br to dst.

if fetchData is false, then only block header is read, otherwise all the data is read.

func (*BlockRef) PartRef added in v1.46.0 

func (br *BlockRef) PartRef() PartRef

PartRef returns PartRef from br.

type IndexDBMetrics 

type IndexDBMetrics struct {
	TagCacheSize      uint64
	TagCacheSizeBytes uint64
	TagCacheRequests  uint64
	TagCacheMisses    uint64

	UselessTagFiltersCacheSize      uint64
	UselessTagFiltersCacheSizeBytes uint64
	UselessTagFiltersCacheRequests  uint64
	UselessTagFiltersCacheMisses    uint64

	DeletedMetricsCount uint64

	IndexDBRefCount uint64

	NewTimeseriesCreated    uint64
	MissingTSIDsForMetricID uint64

	RecentHourMetricIDsSearchCalls uint64
	RecentHourMetricIDsSearchHits  uint64
	DateMetricIDsSearchCalls       uint64
	DateMetricIDsSearchHits        uint64

	DateRangeSearchCalls uint64
	DateRangeSearchHits  uint64

	MissingMetricNamesForMetricID uint64

	IndexBlocksWithMetricIDsProcessed      uint64
	IndexBlocksWithMetricIDsIncorrectOrder uint64

	mergeset.TableMetrics
}

IndexDBMetrics contains essential metrics for indexDB.

type MetricBlockRef added in v1.35.0 

type MetricBlockRef struct {
	// The metric name
	MetricName []byte

	// The block reference. Call BlockRef.MustReadBlock in order to obtain the block.
	BlockRef *BlockRef
}

MetricBlockRef contains reference to time series block for a single metric.

type MetricName 

type MetricName struct {
	MetricGroup []byte

	// Tags are optional. They must be sorted by tag Key for canonical view.
	// Use sortTags method.
	Tags []Tag
}

MetricName reperesents a metric name.

func GetMetricName 

func GetMetricName() *MetricName

GetMetricName returns a MetricName from pool.

func (*MetricName) AddTag 

func (mn *MetricName) AddTag(key, value string)

AddTag adds new tag to mn with the given key and value.

func (*MetricName) AddTagBytes 

func (mn *MetricName) AddTagBytes(key, value []byte)

AddTagBytes adds new tag to mn with the given key and value.

func (*MetricName) CopyFrom 

func (mn *MetricName) CopyFrom(src *MetricName)

CopyFrom copies src to mn.

func (*MetricName) GetTagValue 

func (mn *MetricName) GetTagValue(tagKey string) []byte

GetTagValue returns tag value for the given tagKey.

func (*MetricName) Marshal 

func (mn *MetricName) Marshal(dst []byte) []byte

Marshal appends marshaled mn to dst and returns the result.

mn.sortTags must be called before calling this function in order to sort and de-duplcate tags.

func (*MetricName) RemoveTag 

func (mn *MetricName) RemoveTag(tagKey string)

RemoveTag removes a tag with the given tagKey

func (*MetricName) RemoveTagsIgnoring 

func (mn *MetricName) RemoveTagsIgnoring(ignoringTags []string)

RemoveTagsIgnoring removes all the tags included in ignoringTags.

func (*MetricName) RemoveTagsOn 

func (mn *MetricName) RemoveTagsOn(onTags []string)

RemoveTagsOn removes all the tags not included to onTags.

func (*MetricName) Reset 

func (mn *MetricName) Reset()

Reset resets the mn.

func (*MetricName) ResetMetricGroup 

func (mn *MetricName) ResetMetricGroup()

ResetMetricGroup resets mn.MetricGroup

func (*MetricName) SetTags added in v1.37.3 

func (mn *MetricName) SetTags(addTags []string, src *MetricName)

SetTags sets tags from src with keys matching addTags.

func (*MetricName) String 

func (mn *MetricName) String() string

String returns user-readable representation of the metric name.

func (*MetricName) Unmarshal 

func (mn *MetricName) Unmarshal(src []byte) error

Unmarshal unmarshals mn from src.

type MetricRow 

type MetricRow struct {
	// MetricNameRaw contains raw metric name, which must be decoded
	// with MetricName.unmarshalRaw.
	MetricNameRaw []byte

	Timestamp int64
	Value     float64
}

MetricRow is a metric to insert into storage.

func (*MetricRow) CopyFrom 

func (mr *MetricRow) CopyFrom(src *MetricRow)

CopyFrom copies src to mr.

func (*MetricRow) Marshal 

func (mr *MetricRow) Marshal(dst []byte) []byte

Marshal appends marshaled mr to dst and returns the result.

func (*MetricRow) String 

func (mr *MetricRow) String() string

String returns string representation of the mr.

func (*MetricRow) Unmarshal 

func (mr *MetricRow) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals mr from src and returns the remaining tail from src.

type Metrics 

type Metrics struct {
	RowsAddedTotal    uint64
	DedupsDuringMerge uint64

	TooSmallTimestampRows uint64
	TooBigTimestampRows   uint64

	AddRowsConcurrencyLimitReached uint64
	AddRowsConcurrencyLimitTimeout uint64
	AddRowsConcurrencyDroppedRows  uint64
	AddRowsConcurrencyCapacity     uint64
	AddRowsConcurrencyCurrent      uint64

	SearchTSIDsConcurrencyLimitReached uint64
	SearchTSIDsConcurrencyLimitTimeout uint64
	SearchTSIDsConcurrencyCapacity     uint64
	SearchTSIDsConcurrencyCurrent      uint64

	SearchDelays uint64

	SlowRowInserts         uint64
	SlowPerDayIndexInserts uint64
	SlowMetricNameLoads    uint64

	TimestampsBlocksMerged uint64
	TimestampsBytesSaved   uint64

	TSIDCacheSize       uint64
	TSIDCacheSizeBytes  uint64
	TSIDCacheRequests   uint64
	TSIDCacheMisses     uint64
	TSIDCacheCollisions uint64

	MetricIDCacheSize       uint64
	MetricIDCacheSizeBytes  uint64
	MetricIDCacheRequests   uint64
	MetricIDCacheMisses     uint64
	MetricIDCacheCollisions uint64

	MetricNameCacheSize       uint64
	MetricNameCacheSizeBytes  uint64
	MetricNameCacheRequests   uint64
	MetricNameCacheMisses     uint64
	MetricNameCacheCollisions uint64

	DateMetricIDCacheSize        uint64
	DateMetricIDCacheSizeBytes   uint64
	DateMetricIDCacheSyncsCount  uint64
	DateMetricIDCacheResetsCount uint64

	HourMetricIDCacheSize      uint64
	HourMetricIDCacheSizeBytes uint64

	NextDayMetricIDCacheSize      uint64
	NextDayMetricIDCacheSizeBytes uint64

	PrefetchedMetricIDsSize      uint64
	PrefetchedMetricIDsSizeBytes uint64

	IndexDBMetrics IndexDBMetrics
	TableMetrics   TableMetrics
}

Metrics contains essential metrics for the Storage.

func (*Metrics) Reset 

func (m *Metrics) Reset()

Reset resets m.

type PartRef added in v1.46.0 

type PartRef struct {
	// contains filtered or unexported fields
}

PartRef is Part reference. 

type Search struct {
	// MetricBlockRef is updated with each Search.NextMetricBlock call.
	MetricBlockRef MetricBlockRef
	// contains filtered or unexported fields
}

Search is a search for time series.

func (*Search) Error 

func (s *Search) Error() error

Error returns the last error from s.

func (*Search) Init 

func (s *Search) Init(storage *Storage, tfss []*TagFilters, tr TimeRange, maxMetrics int, deadline uint64) int

Init initializes s from the given storage, tfss and tr.

MustClose must be called when the search is done.

Init returns the upper bound on the number of found time series.

func (*Search) MustClose 

func (s *Search) MustClose()

MustClose closes the Search.

func (*Search) NextMetricBlock 

func (s *Search) NextMetricBlock() bool

NextMetricBlock proceeds to the next MetricBlockRef.

type SearchQuery 

type SearchQuery struct {
	MinTimestamp int64
	MaxTimestamp int64
	TagFilterss  [][]TagFilter
}

SearchQuery is used for sending search queries from vmselect to vmstorage.

func NewSearchQuery added in v1.47.0 

func NewSearchQuery(start, end int64, tagFilterss [][]TagFilter) *SearchQuery

NewSearchQuery creates new search query for the given args.

func (*SearchQuery) Marshal 

func (sq *SearchQuery) Marshal(dst []byte) []byte

Marshal appends marshaled sq to dst and returns the result.

func (*SearchQuery) String 

func (sq *SearchQuery) String() string

String returns string representation of the search query.

func (*SearchQuery) Unmarshal 

func (sq *SearchQuery) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals sq from src and returns the tail.

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage represents TSDB storage.

func OpenStorage 

func OpenStorage(path string, retentionMsecs int64) (*Storage, error)

OpenStorage opens storage on the given path with the given retentionMsecs.

func (*Storage) AddRows 

func (s *Storage) AddRows(mrs []MetricRow, precisionBits uint8) error

AddRows adds the given mrs to s.

func (*Storage) CreateSnapshot 

func (s *Storage) CreateSnapshot() (string, error)

CreateSnapshot creates snapshot for s and returns the snapshot name.

func (*Storage) DebugFlush added in v1.47.0 

func (s *Storage) DebugFlush()

DebugFlush flushes recently added storage data, so it becomes visible to search.

func (*Storage) DeleteMetrics 

func (s *Storage) DeleteMetrics(tfss []*TagFilters) (int, error)

DeleteMetrics deletes all the metrics matching the given tfss.

Returns the number of metrics deleted.

func (*Storage) DeleteSnapshot 

func (s *Storage) DeleteSnapshot(snapshotName string) error

DeleteSnapshot deletes the given snapshot.

func (*Storage) ForceMergePartitions added in v1.41.1 

func (s *Storage) ForceMergePartitions(partitionNamePrefix string) error

ForceMergePartitions force-merges partitions in s with names starting from the given partitionNamePrefix.

Partitions are merged sequentially in order to reduce load on the system.

func (*Storage) GetSeriesCount 

func (s *Storage) GetSeriesCount(deadline uint64) (uint64, error)

GetSeriesCount returns the approximate number of unique time series.

It includes the deleted series too and may count the same series up to two times - in db and extDB.

func (*Storage) GetTSDBStatusForDate added in v1.35.0 

func (s *Storage) GetTSDBStatusForDate(date uint64, topN int, deadline uint64) (*TSDBStatus, error)

GetTSDBStatusForDate returns TSDB status data for /api/v1/status/tsdb.

See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

func (*Storage) ListSnapshots 

func (s *Storage) ListSnapshots() ([]string, error)

ListSnapshots returns sorted list of existing snapshots for s.

func (*Storage) MustClose 

func (s *Storage) MustClose()

MustClose closes the storage.

func (*Storage) RegisterMetricNames added in v1.47.0 

func (s *Storage) RegisterMetricNames(mrs []MetricRow) error

RegisterMetricNames registers all the metric names from mns in the indexdb, so they can be queried later.

The the MetricRow.Timestamp is used for registering the metric name starting from the given timestamp. Th MetricRow.Value field is ignored.

func (*Storage) SearchMetricNames added in v1.47.0 

func (s *Storage) SearchMetricNames(tfss []*TagFilters, tr TimeRange, maxMetrics int, deadline uint64) ([]MetricName, error)

SearchMetricNames returns metric names matching the given tfss on the given tr.

func (*Storage) SearchTagEntries added in v1.19.0 

func (s *Storage) SearchTagEntries(maxTagKeys, maxTagValues int, deadline uint64) ([]TagEntry, error)

SearchTagEntries returns a list of (tagName -> tagValues)

func (*Storage) SearchTagKeys 

func (s *Storage) SearchTagKeys(maxTagKeys int, deadline uint64) ([]string, error)

SearchTagKeys searches for tag keys

func (*Storage) SearchTagKeysOnTimeRange added in v1.46.0 

func (s *Storage) SearchTagKeysOnTimeRange(tr TimeRange, maxTagKeys int, deadline uint64) ([]string, error)

SearchTagKeysOnTimeRange searches for tag keys on tr.

func (*Storage) SearchTagValueSuffixes added in v1.41.0 

func (s *Storage) SearchTagValueSuffixes(tr TimeRange, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int, deadline uint64) ([]string, error)

SearchTagValueSuffixes returns all the tag value suffixes for the given tagKey and tagValuePrefix on the given tr.

This allows implementing https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find or similar APIs.

func (*Storage) SearchTagValues 

func (s *Storage) SearchTagValues(tagKey []byte, maxTagValues int, deadline uint64) ([]string, error)

SearchTagValues searches for tag values for the given tagKey

func (*Storage) SearchTagValuesOnTimeRange added in v1.46.0 

func (s *Storage) SearchTagValuesOnTimeRange(tagKey []byte, tr TimeRange, maxTagValues int, deadline uint64) ([]string, error)

SearchTagValuesOnTimeRange searches for tag values for the given tagKey on tr.

func (*Storage) UpdateMetrics 

func (s *Storage) UpdateMetrics(m *Metrics)

UpdateMetrics updates m with metrics from s.

type TSDBStatus added in v1.35.0 

type TSDBStatus struct {
	SeriesCountByMetricName     []TopHeapEntry
	LabelValueCountByLabelName  []TopHeapEntry
	SeriesCountByLabelValuePair []TopHeapEntry
}

TSDBStatus contains TSDB status data for /api/v1/status/tsdb.

See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

type TSID 

type TSID struct {
	// MetricGroupID is the id of metric group.
	//
	// MetricGroupID must be unique.
	//
	// Metric group contains metrics with the identical name like
	// 'memory_usage', 'http_requests', but with different
	// labels. For instance, the following metrics belong
	// to a metric group 'memory_usage':
	//
	//   memory_usage{datacenter="foo1", job="bar1", instance="baz1:1234"}
	//   memory_usage{datacenter="foo1", job="bar1", instance="baz2:1234"}
	//   memory_usage{datacenter="foo1", job="bar2", instance="baz1:1234"}
	//   memory_usage{datacenter="foo2", job="bar1", instance="baz2:1234"}
	MetricGroupID uint64

	// JobID is the id of an individual job (aka service).
	//
	// JobID must be unique.
	//
	// Service may consist of multiple instances.
	// See https://prometheus.io/docs/concepts/jobs_instances/ for details.
	JobID uint32

	// InstanceID is the id of an instance (aka process).
	//
	// InstanceID must be unique.
	//
	// See https://prometheus.io/docs/concepts/jobs_instances/ for details.
	InstanceID uint32

	// MetricID is the unique id of the metric (time series).
	//
	// All the other TSID fields may be obtained by MetricID.
	MetricID uint64
}

TSID is unique id for a time series.

Time series blocks are sorted by TSID.

All the fields except MetricID are optional. They exist solely for better grouping of related metrics. It is OK if their meaning differ from their naming.

func (*TSID) Less 

func (t *TSID) Less(b *TSID) bool

Less return true if t < b.

func (*TSID) Marshal 

func (t *TSID) Marshal(dst []byte) []byte

Marshal appends marshaled t to dst and returns the result.

func (*TSID) Unmarshal 

func (t *TSID) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals t from src and returns the rest of src.

type TableMetrics 

type TableMetrics struct {
	PartitionsRefCount uint64
	// contains filtered or unexported fields
}

TableMetrics contains essential metrics for the table.

type Tag 

type Tag struct {
	Key   []byte
	Value []byte
}

Tag represents a (key, value) tag for metric.

func (*Tag) Equal added in v1.28.0 

func (tag *Tag) Equal(t *Tag) bool

Equal returns true if tag equals t

func (*Tag) Marshal 

func (tag *Tag) Marshal(dst []byte) []byte

Marshal appends marshaled tag to dst and returns the result.

func (*Tag) Reset added in v1.28.0 

func (tag *Tag) Reset()

Reset resets the tag.

func (*Tag) Unmarshal 

func (tag *Tag) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals tag from src and returns the remaining data from src.

type TagEntry added in v1.19.0 

type TagEntry struct {
	// Key is tagName
	Key string

	// Values contains all the values for Key.
	Values []string
}

TagEntry contains (tagName -> tagValues) mapping

type TagFilter 

type TagFilter struct {
	Key        []byte
	Value      []byte
	IsNegative bool
	IsRegexp   bool
}

TagFilter represents a single tag filter from SearchQuery.

func (*TagFilter) Marshal 

func (tf *TagFilter) Marshal(dst []byte) []byte

Marshal appends marshaled tf to dst and returns the result.

func (*TagFilter) String 

func (tf *TagFilter) String() string

String returns string representation of tf.

func (*TagFilter) Unmarshal 

func (tf *TagFilter) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals tf from src and returns the tail.

type TagFilters 

type TagFilters struct {
	// contains filtered or unexported fields
}

TagFilters represents filters used for filtering tags.

func NewTagFilters 

func NewTagFilters() *TagFilters

NewTagFilters returns new TagFilters.

func (*TagFilters) Add 

func (tfs *TagFilters) Add(key, value []byte, isNegative, isRegexp bool) error

Add adds the given tag filter to tfs.

MetricGroup must be encoded with nil key.

Finalize must be called after tfs is constructed.

func (*TagFilters) Finalize added in v1.34.5 

func (tfs *TagFilters) Finalize() []*TagFilters

Finalize finalizes tfs and may return complementary TagFilters, which must be added to the resulting set of tag filters.

func (*TagFilters) Reset 

func (tfs *TagFilters) Reset()

Reset resets the tf

func (*TagFilters) String 

func (tfs *TagFilters) String() string

String returns human-readable value for tfs.

type TimeRange 

type TimeRange struct {
	MinTimestamp int64
	MaxTimestamp int64
}

TimeRange is time range.

func (*TimeRange) String 

func (tr *TimeRange) String() string

type TopHeapEntry added in v1.35.0 

type TopHeapEntry struct {
	Name  string
	Count uint64
}

TopHeapEntry represents an entry from `top heap` used in stats.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.