Documentation ¶
Index ¶
- Variables
- func DeduplicateSamples(srcTimestamps []int64, srcValues []float64) ([]int64, []float64)
- func MarshalMetricNameRaw(dst []byte, labels []prompb.Label) []byte
- func PutMetricName(mn *MetricName)
- func RegexpCacheMisses() uint64
- func RegexpCacheRequests() uint64
- func RegexpCacheSize() int
- func SetBigMergeWorkersCount(n int)
- func SetFinalMergeDelay(delay time.Duration)
- func SetLogNewSeries(ok bool)
- func SetMaxLabelsPerTimeseries(maxLabels int)
- func SetMinScrapeIntervalForDeduplication(interval time.Duration)
- func SetSmallMergeWorkersCount(n int)
- type Block
- func (b *Block) AppendRowsWithTimeRangeFilter(dstTimestamps []int64, dstValues []float64, tr TimeRange) ([]int64, []float64)
- func (b *Block) CopyFrom(src *Block)
- func (b *Block) Init(tsid *TSID, timestamps, values []int64, scale int16, precisionBits uint8)
- func (b *Block) MarshalData(timestampsBlockOffset, valuesBlockOffset uint64) ([]byte, []byte, []byte)
- func (b *Block) MarshalPortable(dst []byte) []byte
- func (b *Block) Reset()
- func (b *Block) RowsCount() int
- func (b *Block) UnmarshalData() error
- func (b *Block) UnmarshalPortable(src []byte) ([]byte, error)
- type BlockRef
- type IndexDBMetrics
- type MetricBlockRef
- type MetricName
- func (mn *MetricName) AddTag(key, value string)
- func (mn *MetricName) AddTagBytes(key, value []byte)
- func (mn *MetricName) CopyFrom(src *MetricName)
- func (mn *MetricName) GetTagValue(tagKey string) []byte
- func (mn *MetricName) Marshal(dst []byte) []byte
- func (mn *MetricName) RemoveTag(tagKey string)
- func (mn *MetricName) RemoveTagsIgnoring(ignoringTags []string)
- func (mn *MetricName) RemoveTagsOn(onTags []string)
- func (mn *MetricName) Reset()
- func (mn *MetricName) ResetMetricGroup()
- func (mn *MetricName) SetTags(addTags []string, src *MetricName)
- func (mn *MetricName) String() string
- func (mn *MetricName) Unmarshal(src []byte) error
- func (mn *MetricName) UnmarshalRaw(src []byte) error
- type MetricRow
- type Metrics
- type PartRef
- type Search
- type SearchQuery
- type Storage
- func (s *Storage) AddRows(mrs []MetricRow, precisionBits uint8) error
- func (s *Storage) CreateSnapshot() (string, error)
- func (s *Storage) DebugFlush()
- func (s *Storage) DeleteMetrics(tfss []*TagFilters) (int, error)
- func (s *Storage) DeleteSnapshot(snapshotName string) error
- func (s *Storage) ForceMergePartitions(partitionNamePrefix string) error
- func (s *Storage) GetSeriesCount(deadline uint64) (uint64, error)
- func (s *Storage) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN int, deadline uint64) (*TSDBStatus, error)
- func (s *Storage) ListSnapshots() ([]string, error)
- func (s *Storage) MustClose()
- func (s *Storage) RegisterMetricNames(mrs []MetricRow) error
- func (s *Storage) SearchGraphitePaths(tr TimeRange, query []byte, maxPaths int, deadline uint64) ([]string, error)
- func (s *Storage) SearchMetricNames(tfss []*TagFilters, tr TimeRange, maxMetrics int, deadline uint64) ([]MetricName, error)
- func (s *Storage) SearchTagEntries(maxTagKeys, maxTagValues int, deadline uint64) ([]TagEntry, error)
- func (s *Storage) SearchTagKeys(maxTagKeys int, deadline uint64) ([]string, error)
- func (s *Storage) SearchTagKeysOnTimeRange(tr TimeRange, maxTagKeys int, deadline uint64) ([]string, error)
- func (s *Storage) SearchTagValueSuffixes(tr TimeRange, tagKey, tagValuePrefix []byte, delimiter byte, ...) ([]string, error)
- func (s *Storage) SearchTagValues(tagKey []byte, maxTagValues int, deadline uint64) ([]string, error)
- func (s *Storage) SearchTagValuesOnTimeRange(tagKey []byte, tr TimeRange, maxTagValues int, deadline uint64) ([]string, error)
- func (s *Storage) UpdateMetrics(m *Metrics)
- type TSDBStatus
- type TSID
- type TableMetrics
- type Tag
- type TagEntry
- type TagFilter
- type TagFilters
- type TimeRange
- type TopHeapEntry
Constants ¶
This section is empty.
Variables ¶
var ( // MetricsWithDroppedLabels is the number of metrics with at least a single dropped label MetricsWithDroppedLabels uint64 // TooLongLabelNames is the number of too long label names TooLongLabelNames uint64 // TooLongLabelValues is the number of too long label values TooLongLabelValues uint64 )
var ErrDeadlineExceeded = fmt.Errorf("deadline exceeded")
ErrDeadlineExceeded is returned when the request times out.
Functions ¶
func DeduplicateSamples ¶ added in v1.33.0
DeduplicateSamples removes samples from src* if they are closer to each other than minScrapeInterval.
func MarshalMetricNameRaw ¶
MarshalMetricNameRaw marshals labels to dst and returns the result.
The result must be unmarshaled with MetricName.UnmarshalRaw
func RegexpCacheMisses ¶
func RegexpCacheMisses() uint64
RegexpCacheMisses returns the number of cache misses for regexp cache.
func RegexpCacheRequests ¶
func RegexpCacheRequests() uint64
RegexpCacheRequests returns the number of requests to regexp cache.
func RegexpCacheSize ¶
func RegexpCacheSize() int
RegexpCacheSize returns the number of cached regexps for tag filters.
func SetBigMergeWorkersCount ¶ added in v1.28.1
func SetBigMergeWorkersCount(n int)
SetBigMergeWorkersCount sets the maximum number of concurrent mergers for big blocks.
The function must be called before opening or creating any storage.
func SetFinalMergeDelay ¶ added in v1.44.0
SetFinalMergeDelay sets the delay before doing final merge for partitions without newly ingested data.
This function may be called only before Storage initialization.
func SetLogNewSeries ¶ added in v1.56.0
func SetLogNewSeries(ok bool)
SetLogNewSeries updates new series logging.
This function must be called before any calling any storage functions.
func SetMaxLabelsPerTimeseries ¶ added in v1.27.0
func SetMaxLabelsPerTimeseries(maxLabels int)
SetMaxLabelsPerTimeseries sets the limit on the number of labels per each time series.
Superfluous labels are dropped.
func SetMinScrapeIntervalForDeduplication ¶ added in v1.33.1
SetMinScrapeIntervalForDeduplication sets the minimum interval for data points during de-duplication.
De-duplication is disabled if interval is 0.
This function must be called before initializing the storage.
func SetSmallMergeWorkersCount ¶ added in v1.28.1
func SetSmallMergeWorkersCount(n int)
SetSmallMergeWorkersCount sets the maximum number of concurrent mergers for small blocks.
The function must be called before opening or creating any storage.
Types ¶
type Block ¶
type Block struct {
// contains filtered or unexported fields
}
Block represents a block of time series values for a single TSID.
func (*Block) AppendRowsWithTimeRangeFilter ¶ added in v1.42.0
func (b *Block) AppendRowsWithTimeRangeFilter(dstTimestamps []int64, dstValues []float64, tr TimeRange) ([]int64, []float64)
AppendRowsWithTimeRangeFilter filters samples from b according to tr and appends them to dst*.
It is expected that UnmarshalData has been already called on b.
func (*Block) MarshalData ¶
func (b *Block) MarshalData(timestampsBlockOffset, valuesBlockOffset uint64) ([]byte, []byte, []byte)
MarshalData marshals the block into binary representation.
func (*Block) MarshalPortable ¶ added in v1.42.0
MarshalPortable marshals b to dst, so it could be portably migrated to other VictoriaMetrics instance.
The marshaled value must be unmarshaled with UnmarshalPortable function.
func (*Block) UnmarshalData ¶
UnmarshalData unmarshals block data.
type BlockRef ¶ added in v1.35.0
type BlockRef struct {
// contains filtered or unexported fields
}
BlockRef references a Block.
BlockRef is valid only until the corresponding Search is valid, i.e. it becomes invalid after Search.MustClose is called.
func (*BlockRef) MustReadBlock ¶ added in v1.35.0
MustReadBlock reads block from br to dst.
if fetchData is false, then only block header is read, otherwise all the data is read.
type IndexDBMetrics ¶
type IndexDBMetrics struct { TagFiltersCacheSize uint64 TagFiltersCacheSizeBytes uint64 TagFiltersCacheRequests uint64 TagFiltersCacheMisses uint64 DeletedMetricsCount uint64 IndexDBRefCount uint64 NewTimeseriesCreated uint64 MissingTSIDsForMetricID uint64 RecentHourMetricIDsSearchCalls uint64 RecentHourMetricIDsSearchHits uint64 DateRangeSearchCalls uint64 DateRangeSearchHits uint64 GlobalSearchCalls uint64 MissingMetricNamesForMetricID uint64 IndexBlocksWithMetricIDsProcessed uint64 IndexBlocksWithMetricIDsIncorrectOrder uint64 MinTimestampForCompositeIndex uint64 CompositeFilterSuccessConversions uint64 CompositeFilterMissingConversions uint64 mergeset.TableMetrics }
IndexDBMetrics contains essential metrics for indexDB.
type MetricBlockRef ¶ added in v1.35.0
type MetricBlockRef struct { // The metric name MetricName []byte // The block reference. Call BlockRef.MustReadBlock in order to obtain the block. BlockRef *BlockRef }
MetricBlockRef contains reference to time series block for a single metric.
type MetricName ¶
type MetricName struct { MetricGroup []byte // Tags are optional. They must be sorted by tag Key for canonical view. // Use sortTags method. Tags []Tag }
MetricName reperesents a metric name.
func (*MetricName) AddTag ¶
func (mn *MetricName) AddTag(key, value string)
AddTag adds new tag to mn with the given key and value.
func (*MetricName) AddTagBytes ¶
func (mn *MetricName) AddTagBytes(key, value []byte)
AddTagBytes adds new tag to mn with the given key and value.
func (*MetricName) CopyFrom ¶
func (mn *MetricName) CopyFrom(src *MetricName)
CopyFrom copies src to mn.
func (*MetricName) GetTagValue ¶
func (mn *MetricName) GetTagValue(tagKey string) []byte
GetTagValue returns tag value for the given tagKey.
func (*MetricName) Marshal ¶
func (mn *MetricName) Marshal(dst []byte) []byte
Marshal appends marshaled mn to dst and returns the result.
mn.sortTags must be called before calling this function in order to sort and de-duplcate tags.
func (*MetricName) RemoveTag ¶
func (mn *MetricName) RemoveTag(tagKey string)
RemoveTag removes a tag with the given tagKey
func (*MetricName) RemoveTagsIgnoring ¶
func (mn *MetricName) RemoveTagsIgnoring(ignoringTags []string)
RemoveTagsIgnoring removes all the tags included in ignoringTags.
func (*MetricName) RemoveTagsOn ¶
func (mn *MetricName) RemoveTagsOn(onTags []string)
RemoveTagsOn removes all the tags not included to onTags.
func (*MetricName) ResetMetricGroup ¶
func (mn *MetricName) ResetMetricGroup()
ResetMetricGroup resets mn.MetricGroup
func (*MetricName) SetTags ¶ added in v1.37.3
func (mn *MetricName) SetTags(addTags []string, src *MetricName)
SetTags sets tags from src with keys matching addTags.
func (*MetricName) String ¶
func (mn *MetricName) String() string
String returns user-readable representation of the metric name.
func (*MetricName) Unmarshal ¶
func (mn *MetricName) Unmarshal(src []byte) error
Unmarshal unmarshals mn from src.
func (*MetricName) UnmarshalRaw ¶ added in v1.60.0
func (mn *MetricName) UnmarshalRaw(src []byte) error
UnmarshalRaw unmarshals mn encoded with MarshalMetricNameRaw.
type MetricRow ¶
type MetricRow struct { // MetricNameRaw contains raw metric name, which must be decoded // with MetricName.UnmarshalRaw. MetricNameRaw []byte Timestamp int64 Value float64 }
MetricRow is a metric to insert into storage.
type Metrics ¶
type Metrics struct { RowsAddedTotal uint64 DedupsDuringMerge uint64 TooSmallTimestampRows uint64 TooBigTimestampRows uint64 AddRowsConcurrencyLimitReached uint64 AddRowsConcurrencyLimitTimeout uint64 AddRowsConcurrencyDroppedRows uint64 AddRowsConcurrencyCapacity uint64 AddRowsConcurrencyCurrent uint64 SearchTSIDsConcurrencyLimitReached uint64 SearchTSIDsConcurrencyLimitTimeout uint64 SearchTSIDsConcurrencyCapacity uint64 SearchTSIDsConcurrencyCurrent uint64 SearchDelays uint64 SlowRowInserts uint64 SlowPerDayIndexInserts uint64 SlowMetricNameLoads uint64 HourlySeriesLimitRowsDropped uint64 DailySeriesLimitRowsDropped uint64 TimestampsBlocksMerged uint64 TimestampsBytesSaved uint64 TSIDCacheSize uint64 TSIDCacheSizeBytes uint64 TSIDCacheRequests uint64 TSIDCacheMisses uint64 TSIDCacheCollisions uint64 MetricIDCacheSize uint64 MetricIDCacheSizeBytes uint64 MetricIDCacheRequests uint64 MetricIDCacheMisses uint64 MetricIDCacheCollisions uint64 MetricNameCacheSize uint64 MetricNameCacheSizeBytes uint64 MetricNameCacheRequests uint64 MetricNameCacheMisses uint64 MetricNameCacheCollisions uint64 DateMetricIDCacheSize uint64 DateMetricIDCacheSizeBytes uint64 DateMetricIDCacheSyncsCount uint64 DateMetricIDCacheResetsCount uint64 HourMetricIDCacheSize uint64 HourMetricIDCacheSizeBytes uint64 NextDayMetricIDCacheSize uint64 NextDayMetricIDCacheSizeBytes uint64 PrefetchedMetricIDsSize uint64 PrefetchedMetricIDsSizeBytes uint64 IndexDBMetrics IndexDBMetrics TableMetrics TableMetrics }
Metrics contains essential metrics for the Storage.
type PartRef ¶ added in v1.46.0
type PartRef struct {
// contains filtered or unexported fields
}
PartRef is Part reference.
type Search ¶
type Search struct { // MetricBlockRef is updated with each Search.NextMetricBlock call. MetricBlockRef MetricBlockRef // contains filtered or unexported fields }
Search is a search for time series.
func (*Search) Init ¶
func (s *Search) Init(storage *Storage, tfss []*TagFilters, tr TimeRange, maxMetrics int, deadline uint64) int
Init initializes s from the given storage, tfss and tr.
MustClose must be called when the search is done.
Init returns the upper bound on the number of found time series.
func (*Search) NextMetricBlock ¶
NextMetricBlock proceeds to the next MetricBlockRef.
type SearchQuery ¶
SearchQuery is used for sending search queries from vmselect to vmstorage.
func NewSearchQuery ¶ added in v1.47.0
func NewSearchQuery(start, end int64, tagFilterss [][]TagFilter) *SearchQuery
NewSearchQuery creates new search query for the given args.
func (*SearchQuery) Marshal ¶
func (sq *SearchQuery) Marshal(dst []byte) []byte
Marshal appends marshaled sq to dst and returns the result.
func (*SearchQuery) String ¶
func (sq *SearchQuery) String() string
String returns string representation of the search query.
type Storage ¶
type Storage struct {
// contains filtered or unexported fields
}
Storage represents TSDB storage.
func OpenStorage ¶
func OpenStorage(path string, retentionMsecs int64, maxHourlySeries, maxDailySeries int) (*Storage, error)
OpenStorage opens storage on the given path with the given retentionMsecs.
func (*Storage) CreateSnapshot ¶
CreateSnapshot creates snapshot for s and returns the snapshot name.
func (*Storage) DebugFlush ¶ added in v1.47.0
func (s *Storage) DebugFlush()
DebugFlush flushes recently added storage data, so it becomes visible to search.
func (*Storage) DeleteMetrics ¶
func (s *Storage) DeleteMetrics(tfss []*TagFilters) (int, error)
DeleteMetrics deletes all the metrics matching the given tfss.
Returns the number of metrics deleted.
func (*Storage) DeleteSnapshot ¶
DeleteSnapshot deletes the given snapshot.
func (*Storage) ForceMergePartitions ¶ added in v1.41.1
ForceMergePartitions force-merges partitions in s with names starting from the given partitionNamePrefix.
Partitions are merged sequentially in order to reduce load on the system.
func (*Storage) GetSeriesCount ¶
GetSeriesCount returns the approximate number of unique time series.
It includes the deleted series too and may count the same series up to two times - in db and extDB.
func (*Storage) GetTSDBStatusWithFiltersForDate ¶ added in v1.60.0
func (s *Storage) GetTSDBStatusWithFiltersForDate(tfss []*TagFilters, date uint64, topN int, deadline uint64) (*TSDBStatus, error)
GetTSDBStatusWithFiltersForDate returns TSDB status data for /api/v1/status/tsdb with match[] filters.
func (*Storage) ListSnapshots ¶
ListSnapshots returns sorted list of existing snapshots for s.
func (*Storage) MustClose ¶
func (s *Storage) MustClose()
MustClose closes the storage.
It is expected that the s is no longer used during the close.
func (*Storage) RegisterMetricNames ¶ added in v1.47.0
RegisterMetricNames registers all the metric names from mns in the indexdb, so they can be queried later.
The the MetricRow.Timestamp is used for registering the metric name starting from the given timestamp. Th MetricRow.Value field is ignored.
func (*Storage) SearchGraphitePaths ¶ added in v1.53.0
func (s *Storage) SearchGraphitePaths(tr TimeRange, query []byte, maxPaths int, deadline uint64) ([]string, error)
SearchGraphitePaths returns all the matching paths for the given graphite query on the given tr.
func (*Storage) SearchMetricNames ¶ added in v1.47.0
func (s *Storage) SearchMetricNames(tfss []*TagFilters, tr TimeRange, maxMetrics int, deadline uint64) ([]MetricName, error)
SearchMetricNames returns metric names matching the given tfss on the given tr.
func (*Storage) SearchTagEntries ¶ added in v1.19.0
func (s *Storage) SearchTagEntries(maxTagKeys, maxTagValues int, deadline uint64) ([]TagEntry, error)
SearchTagEntries returns a list of (tagName -> tagValues)
func (*Storage) SearchTagKeys ¶
SearchTagKeys searches for tag keys
func (*Storage) SearchTagKeysOnTimeRange ¶ added in v1.46.0
func (s *Storage) SearchTagKeysOnTimeRange(tr TimeRange, maxTagKeys int, deadline uint64) ([]string, error)
SearchTagKeysOnTimeRange searches for tag keys on tr.
func (*Storage) SearchTagValueSuffixes ¶ added in v1.41.0
func (s *Storage) SearchTagValueSuffixes(tr TimeRange, tagKey, tagValuePrefix []byte, delimiter byte, maxTagValueSuffixes int, deadline uint64) ([]string, error)
SearchTagValueSuffixes returns all the tag value suffixes for the given tagKey and tagValuePrefix on the given tr.
This allows implementing https://graphite-api.readthedocs.io/en/latest/api.html#metrics-find or similar APIs.
If more than maxTagValueSuffixes suffixes is found, then only the first maxTagValueSuffixes suffixes is returned.
func (*Storage) SearchTagValues ¶
func (s *Storage) SearchTagValues(tagKey []byte, maxTagValues int, deadline uint64) ([]string, error)
SearchTagValues searches for tag values for the given tagKey
func (*Storage) SearchTagValuesOnTimeRange ¶ added in v1.46.0
func (s *Storage) SearchTagValuesOnTimeRange(tagKey []byte, tr TimeRange, maxTagValues int, deadline uint64) ([]string, error)
SearchTagValuesOnTimeRange searches for tag values for the given tagKey on tr.
func (*Storage) UpdateMetrics ¶
UpdateMetrics updates m with metrics from s.
type TSDBStatus ¶ added in v1.35.0
type TSDBStatus struct { SeriesCountByMetricName []TopHeapEntry LabelValueCountByLabelName []TopHeapEntry SeriesCountByLabelValuePair []TopHeapEntry }
TSDBStatus contains TSDB status data for /api/v1/status/tsdb.
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
type TSID ¶
type TSID struct { // MetricGroupID is the id of metric group. // // MetricGroupID must be unique. // // Metric group contains metrics with the identical name like // 'memory_usage', 'http_requests', but with different // labels. For instance, the following metrics belong // to a metric group 'memory_usage': // // memory_usage{datacenter="foo1", job="bar1", instance="baz1:1234"} // memory_usage{datacenter="foo1", job="bar1", instance="baz2:1234"} // memory_usage{datacenter="foo1", job="bar2", instance="baz1:1234"} // memory_usage{datacenter="foo2", job="bar1", instance="baz2:1234"} MetricGroupID uint64 // JobID is the id of an individual job (aka service). // // JobID must be unique. // // Service may consist of multiple instances. // See https://prometheus.io/docs/concepts/jobs_instances/ for details. JobID uint32 // InstanceID is the id of an instance (aka process). // // InstanceID must be unique. // // See https://prometheus.io/docs/concepts/jobs_instances/ for details. InstanceID uint32 // MetricID is the unique id of the metric (time series). // // All the other TSID fields may be obtained by MetricID. MetricID uint64 }
TSID is unique id for a time series.
Time series blocks are sorted by TSID.
All the fields except MetricID are optional. They exist solely for better grouping of related metrics. It is OK if their meaning differ from their naming.
type TableMetrics ¶
type TableMetrics struct { PartitionsRefCount uint64 // contains filtered or unexported fields }
TableMetrics contains essential metrics for the table.
type Tag ¶
Tag represents a (key, value) tag for metric.
type TagEntry ¶ added in v1.19.0
type TagEntry struct { // Key is tagName Key string // Values contains all the values for Key. Values []string }
TagEntry contains (tagName -> tagValues) mapping
type TagFilter ¶
TagFilter represents a single tag filter from SearchQuery.
type TagFilters ¶
type TagFilters struct {
// contains filtered or unexported fields
}
TagFilters represents filters used for filtering tags.
func (*TagFilters) Add ¶
func (tfs *TagFilters) Add(key, value []byte, isNegative, isRegexp bool) error
Add adds the given tag filter to tfs.
MetricGroup must be encoded with nil key.
func (*TagFilters) AddGraphiteQuery ¶ added in v1.53.0
func (tfs *TagFilters) AddGraphiteQuery(query []byte, paths []string, isNegative bool)
AddGraphiteQuery adds the given Graphite query that matches the given paths to tfs.
func (*TagFilters) String ¶
func (tfs *TagFilters) String() string
String returns human-readable value for tfs.
type TopHeapEntry ¶ added in v1.35.0
TopHeapEntry represents an entry from `top heap` used in stats.
Source Files ¶
- block.go
- block_header.go
- block_stream_merger.go
- block_stream_reader.go
- block_stream_writer.go
- dedup.go
- index_db.go
- inmemory_part.go
- merge.go
- metaindex_row.go
- metric_name.go
- part.go
- part_header.go
- part_search.go
- partition.go
- partition_search.go
- raw_block.go
- raw_row.go
- search.go
- storage.go
- table.go
- table_search.go
- tag_filters.go
- time.go
- tsid.go