logqlengine

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 17, 2023 License: Apache-2.0 Imports: 36 Imported by: 0

Documentation

Overview

Package logqlengine implements LogQL evaluation engine.

Index

Constants

This section is empty.

Variables

View Source
var NopProcessor = &nopProcessor{}

NopProcessor is a processor that does nothing.

Functions

func LineFromRecord added in v0.1.0

func LineFromRecord(record logstorage.Record) string

LineFromRecord returns a JSON line from a log record.

Types

type AndLabelMatcher

type AndLabelMatcher struct {
	Left  Processor
	Right Processor
}

AndLabelMatcher is a AND logical operation for two label matchers.

func (*AndLabelMatcher) Process

func (m *AndLabelMatcher) Process(ts otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type BytesLabelFilter

type BytesLabelFilter[C Comparator[uint64]] struct {
	// contains filtered or unexported fields
}

BytesLabelFilter is a label filter Processor.

func (*BytesLabelFilter[C]) Process

func (lf *BytesLabelFilter[C]) Process(_ otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type Comparator

type Comparator[T any] interface {
	Compare(a, b T) bool
}

Comparator is a filter that compares value.

type ContainsMatcher

type ContainsMatcher struct {
	Value string
}

ContainsMatcher checks if a string contains value.

func (ContainsMatcher) Match

func (m ContainsMatcher) Match(s string) bool

Match implements StringMatcher.

type Decolorize added in v0.1.0

type Decolorize struct{}

Decolorize removes ANSI escape codes from line.

func (*Decolorize) Process added in v0.1.0

func (d *Decolorize) Process(_ otelstorage.Timestamp, line string, _ LabelSet) (string, bool)

Process implements Processor.

type DistinctFilter added in v0.0.8

type DistinctFilter struct {
	// contains filtered or unexported fields
}

DistinctFilter filters out records with duplicate label values.

func (*DistinctFilter) Process added in v0.0.8

func (d *DistinctFilter) Process(_ otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type DropLabels

type DropLabels struct {
	// contains filtered or unexported fields
}

DropLabels label filtering Processor.

func (*DropLabels) Process

func (k *DropLabels) Process(_ otelstorage.Timestamp, line string, set LabelSet) (string, bool)

Process implements Processor.

type DurationLabelFilter

type DurationLabelFilter[C Comparator[time.Duration]] struct {
	// contains filtered or unexported fields
}

DurationLabelFilter is a label filter Processor.

func (*DurationLabelFilter[C]) Process

func (lf *DurationLabelFilter[C]) Process(_ otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a LogQL evaluation engine.

func NewEngine

func NewEngine(querier Querier, opts Options) *Engine

NewEngine creates new Engine.

func (*Engine) Eval

func (e *Engine) Eval(ctx context.Context, query string, params EvalParams) (data lokiapi.QueryResponseData, rerr error)

Eval parses and evaluates query.

type EqComparator

type EqComparator[T comparable] struct{}

EqComparator implements '==' Comparator.

func (EqComparator[T]) Compare

func (EqComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type EqualIPMatcher

type EqualIPMatcher struct {
	Value netip.Addr
}

EqualIPMatcher checks if an IP equal to given value.

func (EqualIPMatcher) Match

func (m EqualIPMatcher) Match(ip netip.Addr) bool

Match implements IPMatcher.

type EqualsMatcher

type EqualsMatcher struct {
	Value string
}

EqualsMatcher checks if a string equals to a value.

func (EqualsMatcher) Match

func (m EqualsMatcher) Match(s string) bool

Match implements StringMatcher.

type EvalParams

type EvalParams struct {
	Start     otelstorage.Timestamp
	End       otelstorage.Timestamp
	Step      time.Duration
	Direction string // forward, backward
	Limit     int
}

EvalParams sets evaluation parameters.

func (EvalParams) IsInstant

func (p EvalParams) IsInstant() bool

IsInstant whether query is instant.

type GtComparator

type GtComparator[T cmp.Ordered] struct{}

GtComparator implements '>' Comparator.

func (GtComparator[T]) Compare

func (GtComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type GteComparator

type GteComparator[T cmp.Ordered] struct{}

GteComparator implements '>=' Comparator.

func (GteComparator[T]) Compare

func (GteComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type IPLabelFilter

type IPLabelFilter struct {
	// contains filtered or unexported fields
}

IPLabelFilter is a label filter Processor.

func (*IPLabelFilter) Process

func (lf *IPLabelFilter) Process(_ otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type IPLineFilter added in v0.0.8

type IPLineFilter struct {
	// contains filtered or unexported fields
}

IPLineFilter looks for IP address in a line and applies matcher to it.

func (*IPLineFilter) Process added in v0.0.8

func (lf *IPLineFilter) Process(_ otelstorage.Timestamp, line string, _ LabelSet) (_ string, keep bool)

Process implements Processor.

type IPMatcher

type IPMatcher interface {
	Matcher[netip.Addr]
}

IPMatcher matches an IP.

type JSONExtractor

type JSONExtractor struct {
	// contains filtered or unexported fields
}

JSONExtractor is a JSON label extractor.

func (*JSONExtractor) Process

func (e *JSONExtractor) Process(_ otelstorage.Timestamp, line string, set LabelSet) (string, bool)

Process implements Processor.

type KeepLabels

type KeepLabels struct {
	// contains filtered or unexported fields
}

KeepLabels label filtering Processor.

func (*KeepLabels) Process

func (k *KeepLabels) Process(_ otelstorage.Timestamp, line string, set LabelSet) (string, bool)

Process implements Processor.

type LabelFormat added in v0.0.8

type LabelFormat struct {
	// contains filtered or unexported fields
}

LabelFormat is a label formatting Processor.

func (*LabelFormat) Process added in v0.0.8

func (lf *LabelFormat) Process(ts otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type LabelMatcher

type LabelMatcher struct {
	// contains filtered or unexported fields
}

LabelMatcher is a label filter Processor.

func (*LabelMatcher) Process

func (lf *LabelMatcher) Process(_ otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type LabelSet

type LabelSet struct {
	// contains filtered or unexported fields
}

LabelSet is a log record's label set.

func (*LabelSet) AsLokiAPI

func (l *LabelSet) AsLokiAPI() lokiapi.LabelSet

AsLokiAPI returns lokiapi.LabelSet

func (*LabelSet) AsMap added in v0.0.8

func (l *LabelSet) AsMap() map[string]string

AsMap returns labels as strings map.

func (*LabelSet) Delete

func (l *LabelSet) Delete(s logql.Label)

Delete deletes label.

func (*LabelSet) Get

func (l *LabelSet) Get(name logql.Label) (v pcommon.Value, ok bool)

Get returns attr value.

func (*LabelSet) GetError

func (l *LabelSet) GetError() (string, bool)

GetError returns error label.

func (*LabelSet) GetFloat added in v0.0.8

func (l *LabelSet) GetFloat(name logql.Label) (_ float64, ok bool, err error)

GetFloat returns number attr value.

func (*LabelSet) GetString

func (l *LabelSet) GetString(name logql.Label) (string, bool)

GetString returns stringified attr value.

func (*LabelSet) Range

func (l *LabelSet) Range(cb func(logql.Label, pcommon.Value))

Range iterates over label set.

func (*LabelSet) Set added in v0.0.8

func (l *LabelSet) Set(s logql.Label, val pcommon.Value)

Set sets label.

func (*LabelSet) SetAttrs

func (l *LabelSet) SetAttrs(attrMaps ...otelstorage.Attrs)

SetAttrs sets labels from attrs.

func (*LabelSet) SetError

func (l *LabelSet) SetError(typ string, err error)

SetError sets special error label.

func (*LabelSet) SetFromRecord

func (l *LabelSet) SetFromRecord(record logstorage.Record)

SetFromRecord sets labels from given log record.

func (*LabelSet) String

func (l *LabelSet) String() string

String returns text representation of labels.

type LineFilter

type LineFilter struct {
	// contains filtered or unexported fields
}

LineFilter is a line matching Processor.

func (*LineFilter) Process

func (lf *LineFilter) Process(_ otelstorage.Timestamp, line string, _ LabelSet) (_ string, keep bool)

Process implements Processor.

type LineFormat added in v0.0.8

type LineFormat struct {
	// contains filtered or unexported fields
}

LineFormat is a line formatting Processor.

func (*LineFormat) Process added in v0.0.8

func (lf *LineFormat) Process(ts otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type LogfmtExtractor

type LogfmtExtractor struct {
	// contains filtered or unexported fields
}

LogfmtExtractor is a Logfmt label extractor.

func (*LogfmtExtractor) Process

func (e *LogfmtExtractor) Process(_ otelstorage.Timestamp, line string, set LabelSet) (string, bool)

Process implements Processor.

type LtComparator

type LtComparator[T cmp.Ordered] struct{}

LtComparator implements '<' Comparator.

func (LtComparator[T]) Compare

func (LtComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type LteComparator

type LteComparator[T cmp.Ordered] struct{}

LteComparator implements '<=' Comparator.

func (LteComparator[T]) Compare

func (LteComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type Matcher

type Matcher[T any] interface {
	Match(T) bool
}

Matcher is a generic matcher.

type NotEqComparator

type NotEqComparator[T comparable] struct{}

NotEqComparator implements '!=' Comparator.

func (NotEqComparator[T]) Compare

func (NotEqComparator[T]) Compare(a, b T) bool

Compare implements Comparator[T].

type NotMatcher

type NotMatcher[T any, M Matcher[T]] struct {
	Next M
}

NotMatcher is a NOT logical matcher.

func (NotMatcher[T, M]) Match

func (m NotMatcher[T, M]) Match(v T) bool

Match implements StringMatcher.

type NumberLabelFilter

type NumberLabelFilter[C Comparator[float64]] struct {
	// contains filtered or unexported fields
}

NumberLabelFilter is a label filter Processor.

func (*NumberLabelFilter[C]) Process

func (lf *NumberLabelFilter[C]) Process(_ otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type Options

type Options struct {
	// LookbackDuration sets lookback duration for instant queries.
	//
	// Should be negative, otherwise default value would be used.
	LookbackDuration time.Duration

	// ParseOptions is a LogQL parser options.
	ParseOptions logql.ParseOptions

	// TracerProvider provides OpenTelemetry tracer for this engine.
	TracerProvider trace.TracerProvider
}

Options sets Engine options.

type OrLabelMatcher

type OrLabelMatcher struct {
	Left  Processor
	Right Processor
}

OrLabelMatcher is a OR logical operation for two label matchers.

func (*OrLabelMatcher) Process

func (m *OrLabelMatcher) Process(ts otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type PatternExtractor added in v0.1.0

type PatternExtractor struct {
	// contains filtered or unexported fields
}

PatternExtractor is a Pattern label extractor.

func (*PatternExtractor) Process added in v0.1.0

func (e *PatternExtractor) Process(_ otelstorage.Timestamp, line string, set LabelSet) (string, bool)

Process implements Processor.

type Pipeline

type Pipeline struct {
	Stages []Processor
}

Pipeline is a multi-stage processor.

func (*Pipeline) Process

func (p *Pipeline) Process(ts otelstorage.Timestamp, line string, attrs LabelSet) (_ string, keep bool)

Process implements Processor.

type PrefixIPMatcher added in v0.0.8

type PrefixIPMatcher struct {
	Prefix netip.Prefix
}

PrefixIPMatcher checks if an IP has given prefix.

func (PrefixIPMatcher) Match added in v0.0.8

func (m PrefixIPMatcher) Match(ip netip.Addr) bool

Match implements IPMatcher.

type Processor

type Processor interface {
	Process(ts otelstorage.Timestamp, line string, labels LabelSet) (newLine string, keep bool)
}

Processor is a log record processor.

func BuildPipeline

func BuildPipeline(stages ...logql.PipelineStage) (Processor, error)

BuildPipeline builds a new Pipeline.

type Querier

type Querier interface {
	// Capabilities returns Querier capabilities.
	//
	// NOTE: engine would call once and then save value.
	// 	Capabilities should not change over time.
	Capabilities() QuerierCapabilities
	// SelectLogs selects log records from storage.
	SelectLogs(ctx context.Context, start, end otelstorage.Timestamp, params SelectLogsParams) (iterators.Iterator[logstorage.Record], error)
}

Querier does queries to storage.

type QuerierCapabilities added in v0.1.0

type QuerierCapabilities struct {
	Label SupportedOps
	Line  SupportedOps
}

QuerierCapabilities defines what operations storage can do.

type RangeIPMatcher

type RangeIPMatcher struct {
	// FIXME(tdakkota): probably, it is better to just use two addrs
	// 	and compare them.
	Range netipx.IPRange
}

RangeIPMatcher checks if an IP is in given range.

func (RangeIPMatcher) Match

func (m RangeIPMatcher) Match(ip netip.Addr) bool

Match implements IPMatcher.

type RegexpExtractor

type RegexpExtractor struct {
	// contains filtered or unexported fields
}

RegexpExtractor is a Regexp label extractor.

func (*RegexpExtractor) Process

func (e *RegexpExtractor) Process(_ otelstorage.Timestamp, line string, set LabelSet) (string, bool)

Process implements Processor.

type RegexpMatcher

type RegexpMatcher struct {
	Re *regexp.Regexp
}

RegexpMatcher checks if a matches regular expression.

func (RegexpMatcher) Match

func (m RegexpMatcher) Match(s string) bool

Match implements StringMatcher.

type RenameLabel added in v0.0.8

type RenameLabel struct {
	// contains filtered or unexported fields
}

RenameLabel is a label renaming Processor.

func (*RenameLabel) Process added in v0.0.8

func (rl *RenameLabel) Process(_ otelstorage.Timestamp, line string, set LabelSet) (_ string, keep bool)

Process implements Processor.

type SelectLogsParams

type SelectLogsParams struct {
	Labels []logql.LabelMatcher
	Line   []logql.LineFilter
}

SelectLogsParams is a storage query params.

type StringMatcher

type StringMatcher interface {
	Matcher[string]
}

StringMatcher matches a string.

type SupportedOps

type SupportedOps uint64

SupportedOps is a bitset defining ops supported by Querier.

func (*SupportedOps) Add

func (caps *SupportedOps) Add(ops ...logql.BinOp)

Add sets capability.

func (SupportedOps) Supports

func (caps SupportedOps) Supports(op logql.BinOp) bool

Supports checks if storage supports given ops.

type UnpackExtractor added in v0.0.8

type UnpackExtractor struct{}

UnpackExtractor extracts log entry fron Promtail `pack`-ed entry.

func (*UnpackExtractor) Process added in v0.0.8

func (e *UnpackExtractor) Process(_ otelstorage.Timestamp, line string, set LabelSet) (string, bool)

Process implements Processor.

type UnsupportedError

type UnsupportedError struct {
	Msg string
}

UnsupportedError is an error that reports unsupported expressions.

func (*UnsupportedError) Error

func (e *UnsupportedError) Error() string

Error implements error.

Directories

Path Synopsis
Package jsonexpr provides JSON extractor expression parser.
Package jsonexpr provides JSON extractor expression parser.
Package logqlmetric provides metric queries implementation.
Package logqlmetric provides metric queries implementation.
Package logqlpattern contains parser for LogQL `pattern` stage pattern.
Package logqlpattern contains parser for LogQL `pattern` stage pattern.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL