logqlengine

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 7, 2024 License: Apache-2.0 Imports: 35 Imported by: 0

Documentation

Overview

Package logqlengine implements LogQL evaluation engine.

Index

Constants

This section is empty.

Variables

View Source 
var NopProcessor = &nopProcessor{}

NopProcessor is a processor that does nothing.

Functions

func LineFromEntry added in v0.10.0 

func LineFromEntry(entry Entry) string

LineFromEntry returns a JSON line from a log record.

func VisitNode added in v0.10.0 

func VisitNode[N Node](root Node, cb func(N) error) error

VisitNode visits nodes of given type.

Types

type AndLabelMatcher 

type AndLabelMatcher struct {
	Left  Processor
	Right Processor
}

AndLabelMatcher is a AND logical operation for two label matchers.

func (*AndLabelMatcher) Process 

func (m *AndLabelMatcher) Process(ts otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type BinOp added in v0.10.0 

type BinOp struct {
	Left, Right MetricNode
	Expr        *logql.BinOpExpr
}

BinOp is a MetricNode implementing binary operation.

func (*BinOp) EvalMetric added in v0.10.0 

func (n *BinOp) EvalMetric(ctx context.Context, params MetricParams) (_ StepIterator, rerr error)

EvalMetric implements [EvalMetric].

func (*BinOp) Traverse added in v0.10.0 

func (n *BinOp) Traverse(cb NodeVisitor) error

Traverse implements Node.

type BytesLabelFilter 

type BytesLabelFilter[C Comparator[uint64]] struct {
	// contains filtered or unexported fields
}

BytesLabelFilter is a label filter Processor.

func (*BytesLabelFilter[C]) Process 

func (lf *BytesLabelFilter[C]) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type Comparator 

type Comparator[T any] interface {
	Compare(a, b T) bool
}

Comparator is a filter that compares value.

type ContainsMatcher 

type ContainsMatcher struct {
	Value string
}

ContainsMatcher checks if a string contains value.

func (ContainsMatcher) Match 

func (m ContainsMatcher) Match(s string) bool

Match implements StringMatcher.

type Decolorize added in v0.1.0 

type Decolorize struct{}

Decolorize removes ANSI escape codes from line.

func (*Decolorize) Process added in v0.1.0 

func (d *Decolorize) Process(_ otelstorage.Timestamp, line string, _ logqlabels.LabelSet) (string, bool)

Process implements Processor.

type Direction added in v0.10.0 

type Direction string

Direction describe log ordering. 

const (
	// DirectionBackward sorts records in descending order.
	DirectionBackward Direction = "backward"
	// DirectionForward sorts records in ascending order.
	DirectionForward Direction = "forward"
)

type DistinctFilter added in v0.0.8 

type DistinctFilter struct {
	// contains filtered or unexported fields
}

DistinctFilter filters out records with duplicate label values.

func (*DistinctFilter) Process added in v0.0.8 

func (d *DistinctFilter) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type DropLabels 

type DropLabels struct {
	// contains filtered or unexported fields
}

DropLabels label filtering Processor.

func (*DropLabels) Process 

func (k *DropLabels) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (string, bool)

Process implements Processor.

type DurationLabelFilter 

type DurationLabelFilter[C Comparator[time.Duration]] struct {
	// contains filtered or unexported fields
}

DurationLabelFilter is a label filter Processor.

func (*DurationLabelFilter[C]) Process 

func (lf *DurationLabelFilter[C]) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a LogQL evaluation engine.

func NewEngine 

func NewEngine(querier Querier, opts Options) *Engine

NewEngine creates new Engine.

func (*Engine) NewQuery added in v0.10.0 

func (e *Engine) NewQuery(ctx context.Context, query string) (Query, error)

NewQuery creates new Query.

type Entry added in v0.10.0 

type Entry struct {
	Timestamp otelstorage.Timestamp
	Line      string
	Set       logqlabels.LabelSet
}

Entry represents a log entry.

type EntryIterator added in v0.10.0 

type EntryIterator = iterators.Iterator[Entry]

EntryIterator represents a LogQL entry log stream.

type EqComparator 

type EqComparator[T comparable] struct{}

EqComparator implements '==' Comparator.

func (EqComparator[T]) Compare 

func (EqComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type EqualIPMatcher 

type EqualIPMatcher struct {
	Value netip.Addr
}

EqualIPMatcher checks if an IP equal to given value.

func (EqualIPMatcher) Match 

func (m EqualIPMatcher) Match(ip netip.Addr) bool

Match implements IPMatcher.

type EqualsMatcher 

type EqualsMatcher struct {
	Value string
}

EqualsMatcher checks if a string equals to a value.

func (EqualsMatcher) Match 

func (m EqualsMatcher) Match(s string) bool

Match implements StringMatcher.

type EvalParams 

type EvalParams struct {
	Start     time.Time
	End       time.Time
	Step      time.Duration
	Direction Direction // forward, backward
	Limit     int
}

EvalParams sets evaluation parameters.

func (EvalParams) IsInstant 

func (p EvalParams) IsInstant() bool

IsInstant whether query is instant.

type GtComparator 

type GtComparator[T cmp.Ordered] struct{}

GtComparator implements '>' Comparator.

func (GtComparator[T]) Compare 

func (GtComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type GteComparator 

type GteComparator[T cmp.Ordered] struct{}

GteComparator implements '>=' Comparator.

func (GteComparator[T]) Compare 

func (GteComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type IPLabelFilter 

type IPLabelFilter struct {
	// contains filtered or unexported fields
}

IPLabelFilter is a label filter Processor.

func (*IPLabelFilter) Process 

func (lf *IPLabelFilter) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type IPLineFilter added in v0.0.8 

type IPLineFilter struct {
	// contains filtered or unexported fields
}

IPLineFilter looks for IP address in a line and applies matcher to it.

func (*IPLineFilter) Process added in v0.0.8 

func (lf *IPLineFilter) Process(_ otelstorage.Timestamp, line string, _ logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type IPMatcher 

type IPMatcher interface {
	Matcher[netip.Addr]
}

IPMatcher matches an IP.

type JSONExtractor 

type JSONExtractor struct {
	// contains filtered or unexported fields
}

JSONExtractor is a JSON label extractor.

func (*JSONExtractor) Process 

func (e *JSONExtractor) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (string, bool)

Process implements Processor.

type KeepLabels 

type KeepLabels struct {
	// contains filtered or unexported fields
}

KeepLabels label filtering Processor.

func (*KeepLabels) Process 

func (k *KeepLabels) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (string, bool)

Process implements Processor.

type LabelFormat added in v0.0.8 

type LabelFormat struct {
	// contains filtered or unexported fields
}

LabelFormat is a label formatting Processor.

func (*LabelFormat) Process added in v0.0.8 

func (lf *LabelFormat) Process(ts otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type LabelMatcher 

type LabelMatcher struct {
	// contains filtered or unexported fields
}

LabelMatcher is a label filter Processor.

func (*LabelMatcher) Process 

func (lf *LabelMatcher) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type LabelReplace added in v0.10.0 

type LabelReplace struct {
	Input MetricNode
	Expr  *logql.LabelReplaceExpr
}

LabelReplace is a MetricNode implementing `label_replace` function.

func (*LabelReplace) EvalMetric added in v0.10.0 

func (n *LabelReplace) EvalMetric(ctx context.Context, params MetricParams) (_ StepIterator, rerr error)

EvalMetric implements [EvalMetric].

func (*LabelReplace) Traverse added in v0.10.0 

func (n *LabelReplace) Traverse(cb NodeVisitor) error

Traverse implements Node.

type LineFilter 

type LineFilter struct {
	// contains filtered or unexported fields
}

LineFilter is a line matching Processor.

func (*LineFilter) Process 

func (lf *LineFilter) Process(_ otelstorage.Timestamp, line string, _ logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type LineFormat added in v0.0.8 

type LineFormat struct {
	// contains filtered or unexported fields
}

LineFormat is a line formatting Processor.

func (*LineFormat) Process added in v0.0.8 

func (lf *LineFormat) Process(ts otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type LiteralBinOp added in v0.10.0 

type LiteralBinOp struct {
	Input           MetricNode
	Literal         float64
	IsLiteralOnLeft bool
	Expr            *logql.BinOpExpr
}

LiteralBinOp is a MetricNode implementing binary operation on literal.

func (*LiteralBinOp) EvalMetric added in v0.10.0 

func (n *LiteralBinOp) EvalMetric(ctx context.Context, params MetricParams) (_ StepIterator, rerr error)

EvalMetric implements [EvalMetric].

func (*LiteralBinOp) Traverse added in v0.10.0 

func (n *LiteralBinOp) Traverse(cb NodeVisitor) error

Traverse implements Node.

type LiteralQuery added in v0.10.0 

type LiteralQuery struct {
	Value float64
}

LiteralQuery is simple literal expression query.

func (*LiteralQuery) Eval added in v0.10.0 

func (q *LiteralQuery) Eval(ctx context.Context, params EvalParams) (data lokiapi.QueryResponseData, _ error)

Eval implements Query.

type LogQuery added in v0.10.0 

type LogQuery struct {
	Root             PipelineNode
	LookbackDuration time.Duration
}

LogQuery represents a log query.

func (*LogQuery) Eval added in v0.10.0 

func (q *LogQuery) Eval(ctx context.Context, params EvalParams) (data lokiapi.QueryResponseData, _ error)

Eval implements Query.

type LogfmtExtractor 

type LogfmtExtractor struct {
	// contains filtered or unexported fields
}

LogfmtExtractor is a Logfmt label extractor.

func (*LogfmtExtractor) Process 

func (e *LogfmtExtractor) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (string, bool)

Process implements Processor.

type LtComparator 

type LtComparator[T cmp.Ordered] struct{}

LtComparator implements '<' Comparator.

func (LtComparator[T]) Compare 

func (LtComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type LteComparator 

type LteComparator[T cmp.Ordered] struct{}

LteComparator implements '<=' Comparator.

func (LteComparator[T]) Compare 

func (LteComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type Matcher 

type Matcher[T any] interface {
	Match(T) bool
}

Matcher is a generic matcher.

type MetricNode added in v0.10.0 

type MetricNode interface {
	Node
	EvalMetric(ctx context.Context, params MetricParams) (StepIterator, error)
}

MetricNode represents a LogQL metric function node.

type MetricParams added in v0.10.0 

type MetricParams struct {
	Start time.Time
	End   time.Time
	Step  time.Duration
}

MetricParams defines MetricNode parameters.

type MetricQuery added in v0.10.0 

type MetricQuery struct {
	Root MetricNode
}

MetricQuery represents a metric query.

func (*MetricQuery) Eval added in v0.10.0 

func (q *MetricQuery) Eval(ctx context.Context, params EvalParams) (lokiapi.QueryResponseData, error)

Eval implements Query.

type Node added in v0.10.0 

type Node interface {
	// Traverse calls given callback on child nodes.
	Traverse(cb NodeVisitor) error
}

Node is a generic node interface.

type NodeVisitor added in v0.10.0 

type NodeVisitor = func(n Node) error

NodeVisitor is a callback to traverse Node.

type NotEqComparator 

type NotEqComparator[T comparable] struct{}

NotEqComparator implements '!=' Comparator.

func (NotEqComparator[T]) Compare 

func (NotEqComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type NotMatcher 

type NotMatcher[T any, M Matcher[T]] struct {
	Next M
}

NotMatcher is a NOT logical matcher.

func (NotMatcher[T, M]) Match 

func (m NotMatcher[T, M]) Match(v T) bool

Match implements StringMatcher.

type NumberLabelFilter 

type NumberLabelFilter[C Comparator[float64]] struct {
	// contains filtered or unexported fields
}

NumberLabelFilter is a label filter Processor.

func (*NumberLabelFilter[C]) Process 

func (lf *NumberLabelFilter[C]) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type Optimizer added in v0.10.0 

type Optimizer interface {
	// Name returns optimizer name.
	Name() string
	Optimize(ctx context.Context, q Query) (Query, error)
}

Optimizer defines an interface for optimizer.

func DefaultOptimizers added in v0.10.0 

func DefaultOptimizers() []Optimizer

DefaultOptimizers returns slice of default [Optimizer]s.

type Options 

type Options struct {
	// LookbackDuration sets lookback duration for instant queries.
	//
	// Should be negative, otherwise default value would be used.
	LookbackDuration time.Duration

	// ParseOptions is a LogQL parser options.
	ParseOptions logql.ParseOptions

	// OTELAdapter enables 'otel adapter' whatever it is.
	OTELAdapter bool

	// Optimizers defines a list of optimiziers to use.
	Optimizers []Optimizer

	// TracerProvider provides OpenTelemetry tracer for this engine.
	TracerProvider trace.TracerProvider
}

Options sets Engine options.

type OrLabelMatcher 

type OrLabelMatcher struct {
	Left  Processor
	Right Processor
}

OrLabelMatcher is a OR logical operation for two label matchers.

func (*OrLabelMatcher) Process 

func (m *OrLabelMatcher) Process(ts otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type PatternExtractor added in v0.1.0 

type PatternExtractor struct {
	// contains filtered or unexported fields
}

PatternExtractor is a Pattern label extractor.

func (*PatternExtractor) Process added in v0.1.0 

func (e *PatternExtractor) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (string, bool)

Process implements Processor.

type Pipeline 

type Pipeline struct {
	Stages []Processor
}

Pipeline is a multi-stage processor.

func (*Pipeline) Process 

func (p *Pipeline) Process(ts otelstorage.Timestamp, line string, attrs logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type PipelineNode added in v0.10.0 

type PipelineNode interface {
	Node
	EvalPipeline(ctx context.Context, params EvalParams) (EntryIterator, error)
}

PipelineNode represents a LogQL pipeline node.

type PrefixIPMatcher added in v0.0.8 

type PrefixIPMatcher struct {
	Prefix netip.Prefix
}

PrefixIPMatcher checks if an IP has given prefix.

func (PrefixIPMatcher) Match added in v0.0.8 

func (m PrefixIPMatcher) Match(ip netip.Addr) bool

Match implements IPMatcher.

type Processor 

type Processor interface {
	Process(ts otelstorage.Timestamp, line string, labels logqlabels.LabelSet) (newLine string, keep bool)
}

Processor is a log record processor.

func BuildPipeline 

func BuildPipeline(stages ...logql.PipelineStage) (Processor, error)

BuildPipeline builds a new Pipeline.

type ProcessorNode added in v0.10.0 

type ProcessorNode struct {
	Input             PipelineNode
	Prefilter         Processor
	Selector          logql.Selector
	Pipeline          []logql.PipelineStage
	EnableOTELAdapter bool
}

ProcessorNode implements PipelineNode.

func (*ProcessorNode) EvalPipeline added in v0.10.0 

func (n *ProcessorNode) EvalPipeline(ctx context.Context, params EvalParams) (_ EntryIterator, rerr error)

EvalPipeline implements PipelineNode.

func (*ProcessorNode) Traverse added in v0.10.0 

func (n *ProcessorNode) Traverse(cb NodeVisitor) error

Traverse implements Node.

type Querier 

type Querier interface {
	// Capabilities returns Querier capabilities.
	//
	// NOTE: engine would call once and then save value.
	// 	Capabilities should not change over time.
	Capabilities() QuerierCapabilities
	// Query creates new [PipelineNode].
	Query(ctx context.Context, selector []logql.LabelMatcher) (PipelineNode, error)
}

Querier does queries to storage.

type QuerierCapabilities added in v0.1.0 

type QuerierCapabilities struct {
	Label SupportedOps
	Line  SupportedOps
}

QuerierCapabilities defines what operations storage can do.

type Query added in v0.10.0 

type Query interface {
	Eval(ctx context.Context, params EvalParams) (lokiapi.QueryResponseData, error)
}

Query is a LogQL query.

type RangeAggregation added in v0.10.0 

type RangeAggregation struct {
	Input SampleNode
	Expr  *logql.RangeAggregationExpr
}

RangeAggregation is a MetricNode implementing range aggregation.

func (*RangeAggregation) EvalMetric added in v0.10.0 

func (n *RangeAggregation) EvalMetric(ctx context.Context, params MetricParams) (_ StepIterator, rerr error)

EvalMetric implements [EvalMetric].

func (*RangeAggregation) Traverse added in v0.10.0 

func (n *RangeAggregation) Traverse(cb NodeVisitor) error

Traverse implements Node.

type RangeIPMatcher 

type RangeIPMatcher struct {
	// FIXME(tdakkota): probably, it is better to just use two addrs
	// 	and compare them.
	Range netipx.IPRange
}

RangeIPMatcher checks if an IP is in given range.

func (RangeIPMatcher) Match 

func (m RangeIPMatcher) Match(ip netip.Addr) bool

Match implements IPMatcher.

type RegexpExtractor 

type RegexpExtractor struct {
	// contains filtered or unexported fields
}

RegexpExtractor is a Regexp label extractor.

func (*RegexpExtractor) Process 

func (e *RegexpExtractor) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (string, bool)

Process implements Processor.

type RegexpMatcher 

type RegexpMatcher struct {
	Re *regexp.Regexp
}

RegexpMatcher checks if a matches regular expression.

func (RegexpMatcher) Match 

func (m RegexpMatcher) Match(s string) bool

Match implements StringMatcher.

type RenameLabel added in v0.0.8 

type RenameLabel struct {
	// contains filtered or unexported fields
}

RenameLabel is a label renaming Processor.

func (*RenameLabel) Process added in v0.0.8 

func (rl *RenameLabel) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (_ string, keep bool)

Process implements Processor.

type SampleIterator added in v0.10.0 

type SampleIterator = iterators.Iterator[logqlmetric.SampledEntry]

SampleIterator represents a samples stream.

type SampleNode added in v0.10.0 

type SampleNode interface {
	Node
	EvalSample(ctx context.Context, params EvalParams) (SampleIterator, error)
}

SampleNode represents a log sampling node.

type SamplingNode added in v0.10.0 

type SamplingNode struct {
	Input PipelineNode
	Expr  *logql.RangeAggregationExpr
}

SamplingNode implements entry sampling.

func (*SamplingNode) EvalSample added in v0.10.0 

func (n *SamplingNode) EvalSample(ctx context.Context, params EvalParams) (SampleIterator, error)

EvalSample implements SampleNode.

func (*SamplingNode) Traverse added in v0.10.0 

func (n *SamplingNode) Traverse(cb NodeVisitor) error

Traverse implements Node.

type StepIterator added in v0.10.0 

type StepIterator = logqlmetric.StepIterator

StepIterator represents a metric stream.

type StringMatcher 

type StringMatcher interface {
	Matcher[string]
}

StringMatcher matches a string.

type SupportedOps 

type SupportedOps uint64

SupportedOps is a bitset defining ops supported by Querier.

func (*SupportedOps) Add 

func (caps *SupportedOps) Add(ops ...logql.BinOp)

Add sets capability.

func (SupportedOps) Supports 

func (caps SupportedOps) Supports(op logql.BinOp) bool

Supports checks if storage supports given ops.

type UnpackExtractor added in v0.0.8 

type UnpackExtractor struct{}

UnpackExtractor extracts log entry fron Promtail `pack`-ed entry.

func (*UnpackExtractor) Process added in v0.0.8 

func (e *UnpackExtractor) Process(_ otelstorage.Timestamp, line string, set logqlabels.LabelSet) (string, bool)

Process implements Processor.

type Vector added in v0.10.0 

type Vector struct {
	Expr *logql.VectorExpr
}

Vector is a MetricNode implementing vector literal.

func (*Vector) EvalMetric added in v0.10.0 

func (n *Vector) EvalMetric(ctx context.Context, params MetricParams) (_ StepIterator, rerr error)

EvalMetric implements [EvalMetric].

func (*Vector) Traverse added in v0.10.0 

func (n *Vector) Traverse(cb NodeVisitor) error

Traverse implements Node.

type VectorAggregation added in v0.10.0 

type VectorAggregation struct {
	Input MetricNode
	Expr  *logql.VectorAggregationExpr
}

VectorAggregation is a MetricNode implementing vector aggregation.

func (*VectorAggregation) EvalMetric added in v0.10.0 

func (n *VectorAggregation) EvalMetric(ctx context.Context, params MetricParams) (_ StepIterator, rerr error)

EvalMetric implements [EvalMetric].

func (*VectorAggregation) Traverse added in v0.10.0 

func (n *VectorAggregation) Traverse(cb NodeVisitor) error

Traverse implements Node.

Source Files

View all Source files

Directories

Path Synopsis
Package jsonexpr provides JSON extractor expression parser.
 Package jsonexpr provides JSON extractor expression parser.
Package logqlabels contains LogQL label utilities.
 Package logqlabels contains LogQL label utilities.
Package logqlerrors defines LogQL engine errors.
 Package logqlerrors defines LogQL engine errors.
Package logqlmetric provides metric queries implementation.
 Package logqlmetric provides metric queries implementation.
Package logqlpattern contains parser for LogQL `pattern` stage pattern.
 Package logqlpattern contains parser for LogQL `pattern` stage pattern.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.