index

Documentation

Index

• func InternalDropTables(ctx context.Context, connection *connection.Connection) error
• func InternalMaxTTLUpdateDelay() time.Duration
• type CassandraIndex
  • func New(ctx context.Context, reg prometheus.Registerer, ...) (*CassandraIndex, error)
  • func (c *CassandraIndex) AllIDs(ctx context.Context, start time.Time, end time.Time) ([]types.MetricID, error)
  • func (c *CassandraIndex) BlockCassandraWrite(ctx context.Context) error
  • func (c *CassandraIndex) Dump(ctx context.Context, w io.Writer) error
  • func (c *CassandraIndex) DumpByExpirationDate(ctx context.Context, w io.Writer, expirationDate time.Time) error
  • func (c *CassandraIndex) DumpByLabels(ctx context.Context, w io.Writer, start, end time.Time, ...) error
  • func (c *CassandraIndex) DumpByPosting(ctx context.Context, w io.Writer, shard time.Time, name string, value string) error
  • func (c *CassandraIndex) DumpByShard(ctx context.Context, w io.Writer, shard time.Time) error
  • func (c *CassandraIndex) InfoByID(ctx context.Context, w io.Writer, id types.MetricID) error
  • func (c *CassandraIndex) InfoGlobal(ctx context.Context, w io.Writer) error
  • func (c *CassandraIndex) InternalCreateMetric(ctx context.Context, start, end time.Time, metrics []labels.Labels, ...) ([]types.MetricID, error)
  • func (c *CassandraIndex) InternalForceExpirationTimestamp(ctx context.Context, value time.Time) error
  • func (c *CassandraIndex) InternalRunOnce(ctx context.Context, now time.Time) bool
  • func (c *CassandraIndex) InternalUpdatePostingShards(ctx context.Context, ids []types.MetricID, labelsList []labels.Labels, ...) error
  • func (c *CassandraIndex) LabelNames(ctx context.Context, start, end time.Time, matchers []*labels.Matcher) ([]string, error)
  • func (c *CassandraIndex) LabelValues(ctx context.Context, start, end time.Time, name string, ...) ([]string, error)
  • func (c *CassandraIndex) LookupIDs(ctx context.Context, requests []types.LookupRequest) ([]types.MetricID, []int64, error)
  • func (c *CassandraIndex) Search(ctx context.Context, queryStart, queryEnd time.Time, ...) (types.MetricsSet, error)
  • func (c *CassandraIndex) Start(_ context.Context) error
  • func (c *CassandraIndex) Stop() error
  • func (c *CassandraIndex) UnblockCassandraWrite(ctx context.Context) error
  • func (c *CassandraIndex) Verifier(output io.Writer) types.IndexVerifier
• type Options

Functions

func InternalDropTables

func InternalDropTables(ctx context.Context, connection *connection.Connection) error

InternalDropTables drop tables used by the index. This should only be used in test & benchmark.

func InternalMaxTTLUpdateDelay

func InternalMaxTTLUpdateDelay() time.Duration

InternalMaxTTLUpdateDelay return the highest delay between TTL update. This should only be used in test & benchmark.

Types

type CassandraIndex

type CassandraIndex struct {
	// contains filtered or unexported fields
}

func New

func New(
	ctx context.Context,
	reg prometheus.Registerer,
	connection *connection.Connection,
	options Options,
	logger zerolog.Logger,
) (*CassandraIndex, error)

New creates a new CassandraIndex object.

func (*CassandraIndex) AllIDs

func (c *CassandraIndex) AllIDs(ctx context.Context, start time.Time, end time.Time) ([]types.MetricID, error)

AllIDs returns all ids stored in the index.

func (*CassandraIndex) BlockCassandraWrite

func (c *CassandraIndex) BlockCassandraWrite(ctx context.Context) error

func (*CassandraIndex) Dump

func (c *CassandraIndex) Dump(ctx context.Context, w io.Writer) error

Dump writes a CSV with all metrics known by this index. The format should not be considered stable and should only be used for debugging.

func (*CassandraIndex) DumpByExpirationDate

func (c *CassandraIndex) DumpByExpirationDate(ctx context.Context, w io.Writer, expirationDate time.Time) error

func (*CassandraIndex) DumpByLabels

func (c *CassandraIndex) DumpByLabels(
	ctx context.Context,
	w io.Writer,
	start, end time.Time,
	matchers []*labels.Matcher,
) error

func (*CassandraIndex) DumpByPosting

func (c *CassandraIndex) DumpByPosting(
	ctx context.Context,
	w io.Writer,
	shard time.Time,
	name string,
	value string,
) error

func (*CassandraIndex) DumpByShard

func (c *CassandraIndex) DumpByShard(ctx context.Context, w io.Writer, shard time.Time) error

func (*CassandraIndex) InfoByID

func (c *CassandraIndex) InfoByID(ctx context.Context, w io.Writer, id types.MetricID) error

func (*CassandraIndex) InfoGlobal

func (c *CassandraIndex) InfoGlobal(ctx context.Context, w io.Writer) error

func (*CassandraIndex) InternalCreateMetric

func (c *CassandraIndex) InternalCreateMetric(
	ctx context.Context,
	start, end time.Time,
	metrics []labels.Labels,
	ids []types.MetricID,
	expirations []time.Time,
	skipPostings bool,
) ([]types.MetricID, error)

InternalCreateMetric create metrics in the index with ID value forced. This should only be used in test & benchmark. The following condition on input must be meet: * no duplicated metrics * no conflict in IDs * labels must be sorted * len(metrics) == len(ids) == len(expirations).

Note: you can also provide "0" for the IDs and have the index allocate an ID for you. But you should only use 0 in such case, because your non-zero ID shouldn't conflict with ID assigned by the index.

Finally you can skip posting updates, but in this case you must write at least one points for each shards. This case is mostly useful to pre-create metrics in bulk.

The metrics will be added in all shards between start & end.

func (*CassandraIndex) InternalForceExpirationTimestamp

func (c *CassandraIndex) InternalForceExpirationTimestamp(ctx context.Context, value time.Time) error

InternalForceExpirationTimestamp will force the state for the most recently processed day of metrics expiration This should only be used in test & benchmark.

func (*CassandraIndex) InternalRunOnce

func (c *CassandraIndex) InternalRunOnce(ctx context.Context, now time.Time) bool

InternalRunOnce run the tasks scheduled by Run, return true if more work is pending. Prefer using Start() than calling InternalRunOnce multiple time. InternalRunOnce is mostly here for squirreldb-cassandra-index-bench program. When using InternalRunOnce you should set option InternalRunOnceCalled to true or you could cause some race condition.

func (*CassandraIndex) InternalUpdatePostingShards

func (c *CassandraIndex) InternalUpdatePostingShards(
	ctx context.Context,
	ids []types.MetricID,
	labelsList []labels.Labels,
	shards [][]int32,
) error

InternalUpdatePostingShards update the time-sharded postings for given metrics. This is useful during test/benchmark when one want to fill index quickly. This assume len(ids) == len(labels) == len(shards). Each element in the list match: * labels[0] is the labels for ids[0]. * shards[0] is the list of shards ids[0] is present.

func (*CassandraIndex) LabelNames

func (c *CassandraIndex) LabelNames(
	ctx context.Context,
	start, end time.Time,
	matchers []*labels.Matcher,
) ([]string, error)

LabelNames returns the unique label names for metrics matching matchers in sorted order. Typical matchers will filter by tenant ID, to get all values for one tenant.

func (*CassandraIndex) LabelValues

func (c *CassandraIndex) LabelValues(
	ctx context.Context,
	start, end time.Time,
	name string,
	matchers []*labels.Matcher,
) ([]string, error)

LabelValues returns potential values for a label name. Values will have at least one metrics matching matchers. Typical matchers will filter by tenant ID, to get all values for one tenant. It is not safe to use the strings beyond the lifefime of the querier.

func (*CassandraIndex) LookupIDs

func (c *CassandraIndex) LookupIDs(
	ctx context.Context,
	requests []types.LookupRequest,
) ([]types.MetricID, []int64, error)

LookupIDs returns a IDs corresponding to the specified labels.Label lists It also return the metric TTLs The result list will be the same length as input lists and using the same order. The input may be mutated (the labels list), so reusing it would be avoided.

func (*CassandraIndex) Search

func (c *CassandraIndex) Search(
	ctx context.Context,
	queryStart, queryEnd time.Time,
	matchers []*labels.Matcher,
) (types.MetricsSet, error)

Search returns a list of IDs corresponding to the specified MetricLabelMatcher list

It implements an inverted index (as used in full-text search). The idea is that word are a couple LabelName=LabelValue. As in an inverted index, it use this "word" to query a posting table which return the list of document ID (metric ID here) that has this "word".

In normal full-text search, the document(s) with the most match will be return, here it return the "document" (metric ID) that exactly match all matchers.

Finally since Matcher could be other thing than LabelName=LabelValue (like not equal or using regular expression) there is a first pass that convert them to something that works with the inverted index: it queries all values for the given label name then for each value, it the value match the filter convert to an simple equal matcher. Then this simple equal matcher could be used with the posting of the inverted index (e.g. name!="cpu" will be converted in something like name="memory" || name="disk" || name="...").

There is still two additional special case: when label should be defined (regardless of the value, e.g. name!="") or when the label should NOT be defined (e.g. name=""). In those case, it use the ability of our posting table to query for metric ID that has a LabelName regardless of the values.

• For label must be defined, it increments the number of Matcher satified if the metric has the label. In principle it's the same as if it expanded it to all possible values (e.g. with name!="" it avoids expanding to name="memory" || name="disk" and directly ask for name=*)
• For label must NOT be defined, it query for all metric IDs that has this label, then increments the number of Matcher satified if currently found metrics are not in the list of metrics having this label. Note: this means that it must already have found some metrics (and that this filter is applied at the end) but PromQL forbids to only have label-not-defined matcher, so some other matcher must exists.

func (*CassandraIndex) Start

func (c *CassandraIndex) Start(_ context.Context) error

Start starts all Cassandra Index services.

func (*CassandraIndex) Stop

func (c *CassandraIndex) Stop() error

Stop stop and wait all Cassandra Index services.

func (*CassandraIndex) UnblockCassandraWrite

func (c *CassandraIndex) UnblockCassandraWrite(ctx context.Context) error

func (*CassandraIndex) Verifier

func (c *CassandraIndex) Verifier(output io.Writer) types.IndexVerifier

type Options

type Options struct {
	LockFactory            lockFactory
	States                 types.State
	SchemaLock             sync.Locker
	Cluster                types.Cluster
	DefaultTimeToLive      time.Duration
	TenantLabelName        string
	PreCreateShardDuration time.Duration
	PreCreateShardFraction int
	ReadOnly               bool
	Logger                 zerolog.Logger
	// InternalRunOnceCalled is true if this index is used by some testings/benchmarking tools
	// that will call RunOnce directly.
	InternalRunOnceCalled bool
	// InternalNowFunction is the function used to get current time. It default to time.Now().
	// This is useful when doing test that need to simulate long period of time.
	InternalNowFunction func() time.Time
}