storage

package
v1.40.1-cluster Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 3, 2020 License: Apache-2.0 Imports: 37 Imported by: 8

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// MetricsWithDroppedLabels is the number of metrics with at least a single dropped label
	MetricsWithDroppedLabels uint64

	// TooLongLabelNames is the number of too long label names
	TooLongLabelNames uint64

	// TooLongLabelValues is the number of too long label values
	TooLongLabelValues uint64
)
View Source 
var ErrDeadlineExceeded = fmt.Errorf("deadline exceeded")

ErrDeadlineExceeded is returned when the request times out.

Functions

func DeduplicateSamples added in v1.33.0 

func DeduplicateSamples(srcTimestamps []int64, srcValues []float64) ([]int64, []float64)

DeduplicateSamples removes samples from src* if they are closer to each other than minScrapeInterval.

func MarshalBlock 

func MarshalBlock(dst []byte, b *Block) []byte

MarshalBlock marshals b to dst.

b.MarshalData must be called on b before calling MarshalBlock.

func MarshalMetricLabelRaw 

func MarshalMetricLabelRaw(dst []byte, label *prompb.Label) []byte

MarshalMetricLabelRaw marshals label to dst.

func MarshalMetricNameRaw 

func MarshalMetricNameRaw(dst []byte, accountID, projectID uint32, labels []prompb.Label) []byte

MarshalMetricNameRaw marshals labels to dst and returns the result.

The result must be unmarshaled with MetricName.unmarshalRaw

func MarshalMetricRow 

func MarshalMetricRow(dst []byte, metricNameRaw []byte, timestamp int64, value float64) []byte

MarshalMetricRow marshals MetricRow data to dst and returns the result.

func PutMetricName 

func PutMetricName(mn *MetricName)

PutMetricName returns mn to the pool.

func RegexpCacheMisses 

func RegexpCacheMisses() uint64

RegexpCacheMisses returns the number of cache misses for regexp cache.

func RegexpCacheRequests 

func RegexpCacheRequests() uint64

RegexpCacheRequests returns the number of requests to regexp cache.

func RegexpCacheSize 

func RegexpCacheSize() int

RegexpCacheSize returns the number of cached regexps for tag filters.

func SetBigMergeWorkersCount added in v1.28.1 

func SetBigMergeWorkersCount(n int)

SetBigMergeWorkersCount sets the maximum number of concurrent mergers for big blocks.

The function must be called before opening or creating any storage.

func SetMaxLabelsPerTimeseries added in v1.27.0 

func SetMaxLabelsPerTimeseries(maxLabels int)

SetMaxLabelsPerTimeseries sets the limit on the number of labels per each time series.

Superfouos labels are dropped.

func SetMinScrapeIntervalForDeduplication added in v1.33.1 

func SetMinScrapeIntervalForDeduplication(interval time.Duration)

SetMinScrapeIntervalForDeduplication sets the minimum interval for data points during de-duplication.

De-duplication is disabled if interval is 0.

This function must be called before initializing the storage.

func SetSmallMergeWorkersCount added in v1.28.1 

func SetSmallMergeWorkersCount(n int)

SetSmallMergeWorkersCount sets the maximum number of concurrent mergers for small blocks.

The function must be called before opening or creating any storage.

func UnmarshalBlock 

func UnmarshalBlock(dst *Block, src []byte) ([]byte, error)

UnmarshalBlock unmarshal Block from src to dst.

dst.UnmarshalData isn't called on the block.

Types

type Block 

type Block struct {
	// contains filtered or unexported fields
}

Block represents a block of time series values for a single TSID.

func (*Block) CopyFrom 

func (b *Block) CopyFrom(src *Block)

CopyFrom copies src to b.

func (*Block) Init 

func (b *Block) Init(tsid *TSID, timestamps, values []int64, scale int16, precisionBits uint8)

Init initializes b with the given tsid, timestamps, values and scale.

func (*Block) MarshalData 

func (b *Block) MarshalData(timestampsBlockOffset, valuesBlockOffset uint64) ([]byte, []byte, []byte)

MarshalData marshals the block into binary representation.

func (*Block) Reset 

func (b *Block) Reset()

Reset resets b.

func (*Block) RowsCount 

func (b *Block) RowsCount() int

RowsCount returns the number of rows in the block.

func (*Block) Scale 

func (b *Block) Scale() int16

Scale returns the decimal scale used for encoding values in the block.

func (*Block) Timestamps 

func (b *Block) Timestamps() []int64

Timestamps returns b timestamps.

func (*Block) UnmarshalData 

func (b *Block) UnmarshalData() error

UnmarshalData unmarshals block data.

func (*Block) Values 

func (b *Block) Values() []int64

Values returns b values.

type BlockRef added in v1.35.0 

type BlockRef struct {
	// contains filtered or unexported fields
}

BlockRef references a Block.

BlockRef is valid only until the corresponding Search is valid, i.e. it becomes invalid after Search.MustClose is called.

func (*BlockRef) MustReadBlock added in v1.35.0 

func (br *BlockRef) MustReadBlock(dst *Block, fetchData bool)

MustReadBlock reads block from br to dst.

if fetchData is false, then only block header is read, otherwise all the data is read.

type IndexDBMetrics 

type IndexDBMetrics struct {
	TagCacheSize      uint64
	TagCacheSizeBytes uint64
	TagCacheRequests  uint64
	TagCacheMisses    uint64

	UselessTagFiltersCacheSize      uint64
	UselessTagFiltersCacheSizeBytes uint64
	UselessTagFiltersCacheRequests  uint64
	UselessTagFiltersCacheMisses    uint64

	DeletedMetricsCount uint64

	IndexDBRefCount uint64

	NewTimeseriesCreated    uint64
	MissingTSIDsForMetricID uint64

	RecentHourMetricIDsSearchCalls uint64
	RecentHourMetricIDsSearchHits  uint64
	DateMetricIDsSearchCalls       uint64
	DateMetricIDsSearchHits        uint64

	DateRangeSearchCalls uint64
	DateRangeSearchHits  uint64

	MissingMetricNamesForMetricID uint64

	IndexBlocksWithMetricIDsProcessed      uint64
	IndexBlocksWithMetricIDsIncorrectOrder uint64

	mergeset.TableMetrics
}

IndexDBMetrics contains essential metrics for indexDB.

type MetricBlock 

type MetricBlock struct {
	// MetricName is metric name for the given Block.
	MetricName []byte

	// Block is a block for the given MetricName
	Block Block
}

MetricBlock is a time series block for a single metric.

func (*MetricBlock) Marshal 

func (mb *MetricBlock) Marshal(dst []byte) []byte

Marshal marshals MetricBlock to dst

func (*MetricBlock) Unmarshal 

func (mb *MetricBlock) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals MetricBlock from src

type MetricBlockRef added in v1.35.0 

type MetricBlockRef struct {
	// The metric name
	MetricName []byte

	// The block reference. Call BlockRef.MustReadBlock in order to obtain the block.
	BlockRef *BlockRef
}

MetricBlockRef contains reference to time series block for a single metric.

type MetricName 

type MetricName struct {
	AccountID uint32
	ProjectID uint32

	MetricGroup []byte

	// Tags are optional. They must be sorted by tag Key for canonical view.
	// Use sortTags method.
	Tags []Tag
}

MetricName reperesents a metric name.

func GetMetricName 

func GetMetricName() *MetricName

GetMetricName returns a MetricName from pool.

func (*MetricName) AddTag 

func (mn *MetricName) AddTag(key, value string)

AddTag adds new tag to mn with the given key and value.

func (*MetricName) AddTagBytes 

func (mn *MetricName) AddTagBytes(key, value []byte)

AddTagBytes adds new tag to mn with the given key and value.

func (*MetricName) CopyFrom 

func (mn *MetricName) CopyFrom(src *MetricName)

CopyFrom copies src to mn.

func (*MetricName) GetTagValue 

func (mn *MetricName) GetTagValue(tagKey string) []byte

GetTagValue returns tag value for the given tagKey.

func (*MetricName) Marshal 

func (mn *MetricName) Marshal(dst []byte) []byte

Marshal appends marshaled mn to dst and returns the result.

Tags must be sorted before calling this function.

func (*MetricName) RemoveTag 

func (mn *MetricName) RemoveTag(tagKey string)

RemoveTag removes a tag with the given tagKey

func (*MetricName) RemoveTagsIgnoring 

func (mn *MetricName) RemoveTagsIgnoring(ignoringTags []string)

RemoveTagsIgnoring removes all the tags included in ignoringTags.

func (*MetricName) RemoveTagsOn 

func (mn *MetricName) RemoveTagsOn(onTags []string)

RemoveTagsOn removes all the tags not included to onTags.

func (*MetricName) Reset 

func (mn *MetricName) Reset()

Reset resets the mn.

func (*MetricName) ResetMetricGroup 

func (mn *MetricName) ResetMetricGroup()

ResetMetricGroup resets mn.MetricGroup

func (*MetricName) SetTags added in v1.37.3 

func (mn *MetricName) SetTags(addTags []string, src *MetricName)

SetTags sets tags from src with keys matching addTags.

func (*MetricName) String 

func (mn *MetricName) String() string

String returns user-readable representation of the metric name.

Use this function only for debug logging.

func (*MetricName) Unmarshal 

func (mn *MetricName) Unmarshal(src []byte) error

Unmarshal unmarshals mn from src.

type MetricRow 

type MetricRow struct {
	// MetricNameRaw contains raw metric name, which must be decoded
	// with MetricName.unmarshalRaw.
	MetricNameRaw []byte

	Timestamp int64
	Value     float64
}

MetricRow is a metric to insert into storage.

func (*MetricRow) CopyFrom 

func (mr *MetricRow) CopyFrom(src *MetricRow)

CopyFrom copies src to mr.

func (*MetricRow) Marshal 

func (mr *MetricRow) Marshal(dst []byte) []byte

Marshal appends marshaled mr to dst and returns the result.

func (*MetricRow) String 

func (mr *MetricRow) String() string

String returns string representation of the mr.

func (*MetricRow) Unmarshal 

func (mr *MetricRow) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals mr from src and returns the remaining tail from src.

type Metrics 

type Metrics struct {
	DedupsDuringMerge uint64

	TooSmallTimestampRows uint64
	TooBigTimestampRows   uint64

	AddRowsConcurrencyLimitReached uint64
	AddRowsConcurrencyLimitTimeout uint64
	AddRowsConcurrencyDroppedRows  uint64
	AddRowsConcurrencyCapacity     uint64
	AddRowsConcurrencyCurrent      uint64

	SearchTSIDsConcurrencyLimitReached uint64
	SearchTSIDsConcurrencyLimitTimeout uint64
	SearchTSIDsConcurrencyCapacity     uint64
	SearchTSIDsConcurrencyCurrent      uint64

	SearchDelays uint64

	SlowRowInserts         uint64
	SlowPerDayIndexInserts uint64
	SlowMetricNameLoads    uint64

	TSIDCacheSize       uint64
	TSIDCacheSizeBytes  uint64
	TSIDCacheRequests   uint64
	TSIDCacheMisses     uint64
	TSIDCacheCollisions uint64

	MetricIDCacheSize       uint64
	MetricIDCacheSizeBytes  uint64
	MetricIDCacheRequests   uint64
	MetricIDCacheMisses     uint64
	MetricIDCacheCollisions uint64

	MetricNameCacheSize       uint64
	MetricNameCacheSizeBytes  uint64
	MetricNameCacheRequests   uint64
	MetricNameCacheMisses     uint64
	MetricNameCacheCollisions uint64

	DateMetricIDCacheSize        uint64
	DateMetricIDCacheSizeBytes   uint64
	DateMetricIDCacheSyncsCount  uint64
	DateMetricIDCacheResetsCount uint64

	HourMetricIDCacheSize      uint64
	HourMetricIDCacheSizeBytes uint64

	NextDayMetricIDCacheSize      uint64
	NextDayMetricIDCacheSizeBytes uint64

	PrefetchedMetricIDsSize      uint64
	PrefetchedMetricIDsSizeBytes uint64

	IndexDBMetrics IndexDBMetrics
	TableMetrics   TableMetrics
}

Metrics contains essential metrics for the Storage.

func (*Metrics) Reset 

func (m *Metrics) Reset()

Reset resets m. 

type Search struct {
	// MetricBlockRef is updated with each Search.NextMetricBlock call.
	MetricBlockRef MetricBlockRef
	// contains filtered or unexported fields
}

Search is a search for time series.

func (*Search) Error 

func (s *Search) Error() error

Error returns the last error from s.

func (*Search) Init 

func (s *Search) Init(storage *Storage, tfss []*TagFilters, tr TimeRange, maxMetrics int, deadline uint64) int

Init initializes s from the given storage, tfss and tr.

MustClose must be called when the search is done.

Init returns the upper bound on the number of found time series.

func (*Search) MustClose 

func (s *Search) MustClose()

MustClose closes the Search.

func (*Search) NextMetricBlock 

func (s *Search) NextMetricBlock() bool

NextMetricBlock proceeds to the next MetricBlockRef.

type SearchQuery 

type SearchQuery struct {
	AccountID    uint32
	ProjectID    uint32
	MinTimestamp int64
	MaxTimestamp int64
	TagFilterss  [][]TagFilter
}

SearchQuery is used for sending search queries from vmselect to vmstorage.

func (*SearchQuery) Marshal 

func (sq *SearchQuery) Marshal(dst []byte) []byte

Marshal appends marshaled sq to dst and returns the result.

func (*SearchQuery) String 

func (sq *SearchQuery) String() string

String returns string representation of the search query.

func (*SearchQuery) Unmarshal 

func (sq *SearchQuery) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals sq from src and returns the tail.

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage represents TSDB storage.

func OpenStorage 

func OpenStorage(path string, retentionMonths int) (*Storage, error)

OpenStorage opens storage on the given path with the given number of retention months.

func (*Storage) AddRows 

func (s *Storage) AddRows(mrs []MetricRow, precisionBits uint8) error

AddRows adds the given mrs to s.

func (*Storage) CreateSnapshot 

func (s *Storage) CreateSnapshot() (string, error)

CreateSnapshot creates snapshot for s and returns the snapshot name.

func (*Storage) DeleteMetrics 

func (s *Storage) DeleteMetrics(tfss []*TagFilters) (int, error)

DeleteMetrics deletes all the metrics matching the given tfss.

Returns the number of metrics deleted.

func (*Storage) DeleteSnapshot 

func (s *Storage) DeleteSnapshot(snapshotName string) error

DeleteSnapshot deletes the given snapshot.

func (*Storage) GetSeriesCount 

func (s *Storage) GetSeriesCount(accountID, projectID uint32, deadline uint64) (uint64, error)

GetSeriesCount returns the approximate number of unique time series for the given (accountID, projectID).

It includes the deleted series too and may count the same series up to two times - in db and extDB.

func (*Storage) GetTSDBStatusForDate added in v1.35.0 

func (s *Storage) GetTSDBStatusForDate(accountID, projectID uint32, date uint64, topN int, deadline uint64) (*TSDBStatus, error)

GetTSDBStatusForDate returns TSDB status data for /api/v1/status/tsdb for the given (accountID, projectID).

See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

func (*Storage) ListSnapshots 

func (s *Storage) ListSnapshots() ([]string, error)

ListSnapshots returns sorted list of existing snapshots for s.

func (*Storage) MustClose 

func (s *Storage) MustClose()

MustClose closes the storage.

func (*Storage) RetentionMonths 

func (s *Storage) RetentionMonths() int

RetentionMonths returns retention months for s.

func (*Storage) SearchTagEntries added in v1.19.0 

func (s *Storage) SearchTagEntries(accountID, projectID uint32, maxTagKeys, maxTagValues int, deadline uint64) ([]TagEntry, error)

SearchTagEntries returns a list of (tagName -> tagValues) for (accountID, projectID).

func (*Storage) SearchTagKeys 

func (s *Storage) SearchTagKeys(accountID, projectID uint32, maxTagKeys int, deadline uint64) ([]string, error)

SearchTagKeys searches for tag keys for the given (accountID, projectID).

func (*Storage) SearchTagValues 

func (s *Storage) SearchTagValues(accountID, projectID uint32, tagKey []byte, maxTagValues int, deadline uint64) ([]string, error)

SearchTagValues searches for tag values for the given tagKey in (accountID, projectID).

func (*Storage) UpdateMetrics 

func (s *Storage) UpdateMetrics(m *Metrics)

UpdateMetrics updates m with metrics from s.

type TSDBStatus added in v1.35.0 

type TSDBStatus struct {
	SeriesCountByMetricName     []TopHeapEntry
	LabelValueCountByLabelName  []TopHeapEntry
	SeriesCountByLabelValuePair []TopHeapEntry
}

TSDBStatus contains TSDB status data for /api/v1/status/tsdb.

See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

type TSID 

type TSID struct {
	// AccountID is the id of the registered account.
	AccountID uint32

	// ProjectID is the id of the project.
	//
	// The ProjectID must be unique for the given AccountID.
	ProjectID uint32

	// MetricGroupID is the id of metric group inside the given project.
	//
	// MetricGroupID must be unique for the given (AccountID, ProjectID).
	//
	// Metric group contains metrics with the identical name like
	// 'memory_usage', 'http_requests', but with different
	// labels. For instance, the following metrics belong
	// to a metric group 'memory_usage':
	//
	//   memory_usage{datacenter="foo1", job="bar1", instance="baz1:1234"}
	//   memory_usage{datacenter="foo1", job="bar1", instance="baz2:1234"}
	//   memory_usage{datacenter="foo1", job="bar2", instance="baz1:1234"}
	//   memory_usage{datacenter="foo2", job="bar1", instance="baz2:1234"}
	MetricGroupID uint64

	// JobID is the id of an individual job (aka service)
	// for the given project.
	//
	// JobID must be unique for the given (AccountID, ProjectID).
	//
	// Service may consist of multiple instances.
	// See https://prometheus.io/docs/concepts/jobs_instances/ for details.
	JobID uint32

	// InstanceID is the id of an instance (aka process)
	// for the given project.
	//
	// InstanceID must be unique for the given (AccountID, ProjectID).
	//
	// See https://prometheus.io/docs/concepts/jobs_instances/ for details.
	InstanceID uint32

	// MetricID is the unique id of the metric (time series).
	//
	// All the other TSID fields may be obtained by MetricID.
	MetricID uint64
}

TSID is unique id for a time series.

Time series blocks are sorted by TSID.

All the fields except MetricID are optional. They exist solely for better grouping of related metrics. It is OK if their meaning differ from their naming.

func (*TSID) Less 

func (t *TSID) Less(b *TSID) bool

Less return true if t < b.

func (*TSID) Marshal 

func (t *TSID) Marshal(dst []byte) []byte

Marshal appends marshaled t to dst and returns the result.

func (*TSID) Unmarshal 

func (t *TSID) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals t from src and returns the rest of src.

type TableMetrics 

type TableMetrics struct {
	PartitionsRefCount uint64
	// contains filtered or unexported fields
}

TableMetrics contains essential metrics for the table.

type Tag 

type Tag struct {
	Key   []byte
	Value []byte
}

Tag represents a (key, value) tag for metric.

func (*Tag) Equal added in v1.28.0 

func (tag *Tag) Equal(t *Tag) bool

Equal returns true if tag equals t

func (*Tag) Marshal 

func (tag *Tag) Marshal(dst []byte) []byte

Marshal appends marshaled tag to dst and returns the result.

func (*Tag) Reset added in v1.28.0 

func (tag *Tag) Reset()

Reset resets the tag.

func (*Tag) Unmarshal 

func (tag *Tag) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals tag from src and returns the remaining data from src.

type TagEntry added in v1.19.0 

type TagEntry struct {
	// Key is tagName
	Key string

	// Values contains all the values for Key.
	Values []string
}

TagEntry contains (tagName -> tagValues) mapping

type TagFilter 

type TagFilter struct {
	Key        []byte
	Value      []byte
	IsNegative bool
	IsRegexp   bool
}

TagFilter represents a single tag filter from SearchQuery.

func (*TagFilter) Marshal 

func (tf *TagFilter) Marshal(dst []byte) []byte

Marshal appends marshaled tf to dst and returns the result.

func (*TagFilter) String 

func (tf *TagFilter) String() string

String returns string representation of tf.

func (*TagFilter) Unmarshal 

func (tf *TagFilter) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals tf from src and returns the tail.

type TagFilters 

type TagFilters struct {
	// contains filtered or unexported fields
}

TagFilters represents filters used for filtering tags.

func NewTagFilters 

func NewTagFilters(accountID, projectID uint32) *TagFilters

NewTagFilters returns new TagFilters for the given accountID and projectID.

func (*TagFilters) Add 

func (tfs *TagFilters) Add(key, value []byte, isNegative, isRegexp bool) error

Add adds the given tag filter to tfs.

MetricGroup must be encoded with nil key.

Finalize must be called after tfs is constructed.

func (*TagFilters) Finalize added in v1.34.5 

func (tfs *TagFilters) Finalize() []*TagFilters

Finalize finalizes tfs and may return complementary TagFilters, which must be added to the resulting set of tag filters.

func (*TagFilters) Reset 

func (tfs *TagFilters) Reset(accountID, projectID uint32)

Reset resets the tf for the given accountID and projectID

func (*TagFilters) String 

func (tfs *TagFilters) String() string

String returns human-readable value for tfs.

type TimeRange 

type TimeRange struct {
	MinTimestamp int64
	MaxTimestamp int64
}

TimeRange is time range.

func (*TimeRange) String 

func (tr *TimeRange) String() string

type TopHeapEntry added in v1.35.0 

type TopHeapEntry struct {
	Name  string
	Count uint64
}

TopHeapEntry represents an entry from `top heap` used in stats.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.