log

package
v3.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 17, 2024 License: AGPL-3.0 Imports: 33 Imported by: 2

Documentation

Index

Constants

View Source 
const (
	IPv4Charset = "0123456789."
	IPv6Charset = "0123456789abcdefABCDEF:."
)
View Source 
const (
	ConvertBytes    = "bytes"
	ConvertDuration = "duration"
	ConvertFloat    = "float"
)
View Source 
const MaxInternedStrings = 1024

Variables

View Source 
var (
	ErrIPFilterInvalidPattern   = errors.New("ip: invalid pattern")
	ErrIPFilterInvalidOperation = errors.New("ip: invalid operation")
)
View Source 
var EmptyLabelsResult = NewLabelsResult(labels.EmptyLabels().String(), labels.EmptyLabels().Hash(), labels.EmptyLabels(), labels.EmptyLabels(), labels.EmptyLabels())
View Source 
var ExistsFilter = existsFilter{}

ExistsFilter is a filter that returns and matches when a line has any characters.

View Source 
var TrueFilter = trueFilter{}

TrueFilter is a filter that returns and matches all log lines whatever their content.

Functions

func IsNoopPipeline 

func IsNoopPipeline(p Pipeline) bool

IsNoopPipeline tells if a pipeline is a Noop.

Types

type AnalyzablePipeline 

type AnalyzablePipeline interface {
	Pipeline
	Stages() []Stage
	LabelsBuilder() *BaseLabelsBuilder
}

type BaseLabelsBuilder 

type BaseLabelsBuilder struct {
	// contains filtered or unexported fields
}

BaseLabelsBuilder is a label builder used by pipeline and stages. Only one base builder is used and it contains cache for each LabelsBuilders.

func NewBaseLabelsBuilder 

func NewBaseLabelsBuilder() *BaseLabelsBuilder

NewBaseLabelsBuilder creates a new base labels builder.

func NewBaseLabelsBuilderWithGrouping 

func NewBaseLabelsBuilderWithGrouping(groups []string, parserKeyHints ParserHint, without, noLabels bool) *BaseLabelsBuilder

NewBaseLabelsBuilderWithGrouping creates a new base labels builder with grouping to compute results.

func (*BaseLabelsBuilder) ForLabels 

func (b *BaseLabelsBuilder) ForLabels(lbs labels.Labels, hash uint64) *LabelsBuilder

ForLabels creates a labels builder for a given labels set as base. The labels cache is shared across all created LabelsBuilders.

func (BaseLabelsBuilder) Hash 

func (h BaseLabelsBuilder) Hash(lbs labels.Labels) uint64

Hash hashes the labels

func (*BaseLabelsBuilder) ParserLabelHints 

func (b *BaseLabelsBuilder) ParserLabelHints() ParserHint

ParserLabelHints returns a limited list of expected labels to extract for metric queries. Returns nil when it's impossible to hint labels extractions.

func (*BaseLabelsBuilder) Reset 

func (b *BaseLabelsBuilder) Reset()

Reset clears all current state for the builder.

type BinaryLabelFilter 

type BinaryLabelFilter struct {
	Left  LabelFilterer
	Right LabelFilterer
	And   bool
}

func NewAndLabelFilter 

func NewAndLabelFilter(left LabelFilterer, right LabelFilterer) *BinaryLabelFilter

NewAndLabelFilter creates a new LabelFilterer from a and binary operation of two LabelFilterer.

func NewOrLabelFilter 

func NewOrLabelFilter(left LabelFilterer, right LabelFilterer) *BinaryLabelFilter

NewOrLabelFilter creates a new LabelFilterer from a or binary operation of two LabelFilterer.

func (*BinaryLabelFilter) Process 

func (b *BinaryLabelFilter) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*BinaryLabelFilter) RequiredLabelNames 

func (b *BinaryLabelFilter) RequiredLabelNames() []string

func (*BinaryLabelFilter) String 

func (b *BinaryLabelFilter) String() string

type BytesLabelFilter 

type BytesLabelFilter struct {
	Name  string
	Value uint64
	Type  LabelFilterType
}

func NewBytesLabelFilter 

func NewBytesLabelFilter(t LabelFilterType, name string, b uint64) *BytesLabelFilter

NewBytesLabelFilter creates a new label filterer which parses bytes string representation (1KB) from the value of the named label and compares it with the given b value.

func (*BytesLabelFilter) Process 

func (d *BytesLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*BytesLabelFilter) RequiredLabelNames 

func (d *BytesLabelFilter) RequiredLabelNames() []string

func (*BytesLabelFilter) String 

func (d *BytesLabelFilter) String() string

type Checker 

type Checker interface {
	Test(line []byte, caseInsensitive bool, equal bool) bool
	TestRegex(reg *regexp.Regexp) bool
}

Checker is an interface that matches against the input line or regexp.

type Decolorizer 

type Decolorizer struct{}

func NewDecolorizer 

func NewDecolorizer() (*Decolorizer, error)

func (Decolorizer) Process 

func (Decolorizer) Process(_ int64, line []byte, _ *LabelsBuilder) ([]byte, bool)

func (Decolorizer) RequiredLabelNames 

func (Decolorizer) RequiredLabelNames() []string

type DropLabel 

type DropLabel struct {
	Matcher *labels.Matcher
	Name    string
}

func NewDropLabel 

func NewDropLabel(matcher *labels.Matcher, name string) DropLabel

type DropLabels 

type DropLabels struct {
	// contains filtered or unexported fields
}

func NewDropLabels 

func NewDropLabels(dl []DropLabel) *DropLabels

func (*DropLabels) Process 

func (dl *DropLabels) Process(_ int64, line []byte, lbls *LabelsBuilder) ([]byte, bool)

func (*DropLabels) RequiredLabelNames 

func (dl *DropLabels) RequiredLabelNames() []string

type DurationLabelFilter 

type DurationLabelFilter struct {
	Name  string
	Value time.Duration
	Type  LabelFilterType
}

func NewDurationLabelFilter 

func NewDurationLabelFilter(t LabelFilterType, name string, d time.Duration) *DurationLabelFilter

NewDurationLabelFilter creates a new label filterer which parses duration string representation (5s) from the value of the named label and compares it with the given d value.

func (*DurationLabelFilter) Process 

func (d *DurationLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*DurationLabelFilter) RequiredLabelNames 

func (d *DurationLabelFilter) RequiredLabelNames() []string

func (*DurationLabelFilter) String 

func (d *DurationLabelFilter) String() string

type Filterer 

type Filterer interface {
	Filter(line []byte) bool
	ToStage() Stage
}

Filterer is a interface to filter log lines.

func ChainOrFilter 

func ChainOrFilter(curr, new Filterer) Filterer

ChainOrFilter is a syntax sugar to chain multiple `or` filters. (1 or many)

func NewAndFilters 

func NewAndFilters(filters []Filterer) Filterer

NewAndFilters creates a new filter which matches only if all filters match

func NewFilter 

func NewFilter(match string, mt LineMatchType) (Filterer, error)

NewFilter creates a new line filter from a match string and type.

func NewLabelFilter 

func NewLabelFilter(match string, mt labels.MatchType) (Filterer, error)

NewLabelFilter creates a new filter that has label regex semantics

type FiltererFunc 

type FiltererFunc func(line []byte) bool

LineFilterFunc is a syntax sugar for creating line filter from a function

func (FiltererFunc) Filter 

func (f FiltererFunc) Filter(line []byte) bool

type Hints 

type Hints struct {
	// contains filtered or unexported fields
}

func NewParserHint 

func NewParserHint(requiredLabelNames, groups []string, without, noLabels bool, metricLabelName string, stages []Stage) *Hints

NewParserHint creates a new parser hint using the list of labels that are seen and required in a query.

func (*Hints) AllRequiredExtracted 

func (p *Hints) AllRequiredExtracted() bool

func (*Hints) NoLabels 

func (p *Hints) NoLabels() bool

func (*Hints) PreserveError 

func (p *Hints) PreserveError() bool

func (*Hints) RecordExtracted 

func (p *Hints) RecordExtracted(key string)

func (*Hints) Reset 

func (p *Hints) Reset()

func (*Hints) ShouldContinueParsingLine 

func (p *Hints) ShouldContinueParsingLine(labelName string, lbs *LabelsBuilder) bool

func (*Hints) ShouldExtract 

func (p *Hints) ShouldExtract(key string) bool

func (*Hints) ShouldExtractPrefix 

func (p *Hints) ShouldExtractPrefix(prefix string) bool

type IPLabelFilter 

type IPLabelFilter struct {
	Ty LabelFilterType

	// if used as Label matcher, this holds the identifier Label name.
	// e.g: (|remote_addr = ip("xxx")). Here labelName is `remote_addr`
	Label string

	// local copy of Pattern to display it in errors, even though Pattern matcher fails because of invalid Pattern.
	Pattern string
	// contains filtered or unexported fields
}

func NewIPLabelFilter 

func NewIPLabelFilter(pattern, label string, ty LabelFilterType) *IPLabelFilter

NewIPLabelFilter is used to construct ip filter as label filter for the given `label`.

func (*IPLabelFilter) PatternError 

func (f *IPLabelFilter) PatternError() error

PatternError will be used `labelFilter.Stage()` method so that, if the given pattern is wrong it returns proper 400 error to the client of LogQL.

func (*IPLabelFilter) Process 

func (f *IPLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

`Process` implements `Stage` interface

func (*IPLabelFilter) RequiredLabelNames 

func (f *IPLabelFilter) RequiredLabelNames() []string

`RequiredLabelNames` implements `Stage` interface

func (*IPLabelFilter) String 

func (f *IPLabelFilter) String() string

`String` implements fmt.Stringer inteface, by which also implements `LabelFilterer` inteface.

type IPLineFilter 

type IPLineFilter struct {
	// contains filtered or unexported fields
}

func NewIPLineFilter 

func NewIPLineFilter(pattern string, ty LineMatchType) (*IPLineFilter, error)

NewIPLineFilter is used to construct ip filter as a `LineFilter`

func (*IPLineFilter) Filter 

func (f *IPLineFilter) Filter(line []byte) bool

Filter implement `Filterer` interface. Used by `LineFilter`

func (*IPLineFilter) Process 

func (f *IPLineFilter) Process(_ int64, line []byte, _ *LabelsBuilder) ([]byte, bool)

`Process` implements `Stage` interface

func (*IPLineFilter) RequiredLabelNames 

func (f *IPLineFilter) RequiredLabelNames() []string

`RequiredLabelNames` implements `Stage` interface

func (*IPLineFilter) ToStage 

func (f *IPLineFilter) ToStage() Stage

ToStage implements `Filterer` interface.

type IPMatchType 

type IPMatchType int

type IPMatcher 

type IPMatcher interface{}

Should be one of the netip.Addr, netip.Prefix, netipx.IPRange.

type JSONExpressionParser 

type JSONExpressionParser struct {
	// contains filtered or unexported fields
}

func NewJSONExpressionParser 

func NewJSONExpressionParser(expressions []LabelExtractionExpr) (*JSONExpressionParser, error)

func (*JSONExpressionParser) Process 

func (j *JSONExpressionParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*JSONExpressionParser) RequiredLabelNames 

func (j *JSONExpressionParser) RequiredLabelNames() []string

type JSONParser 

type JSONParser struct {
	// contains filtered or unexported fields
}

func NewJSONParser 

func NewJSONParser() *JSONParser

NewJSONParser creates a log stage that can parse a json log line and add properties as labels.

func (*JSONParser) Process 

func (j *JSONParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*JSONParser) RequiredLabelNames 

func (j *JSONParser) RequiredLabelNames() []string

type KeepLabel 

type KeepLabel struct {
	Matcher *labels.Matcher
	Name    string
}

func NewKeepLabel 

func NewKeepLabel(matcher *labels.Matcher, name string) KeepLabel

type KeepLabels 

type KeepLabels struct {
	// contains filtered or unexported fields
}

func NewKeepLabels 

func NewKeepLabels(kl []KeepLabel) *KeepLabels

func (*KeepLabels) Process 

func (kl *KeepLabels) Process(_ int64, line []byte, lbls *LabelsBuilder) ([]byte, bool)

func (*KeepLabels) RequiredLabelNames 

func (kl *KeepLabels) RequiredLabelNames() []string

type LabelCategory 

type LabelCategory int
const (
	StreamLabel LabelCategory = iota
	StructuredMetadataLabel
	ParsedLabel
	InvalidCategory
)

type LabelExtractionExpr 

type LabelExtractionExpr struct {
	Identifier string
	Expression string
}

func NewLabelExtractionExpr 

func NewLabelExtractionExpr(identifier, expression string) LabelExtractionExpr

type LabelFilterType 

type LabelFilterType int

LabelFilterType is an enum for label filtering types. 

const (
	LabelFilterEqual LabelFilterType = iota
	LabelFilterNotEqual
	LabelFilterGreaterThan
	LabelFilterGreaterThanOrEqual
	LabelFilterLesserThan
	LabelFilterLesserThanOrEqual
)

Possible LabelFilterType.

func (LabelFilterType) String 

func (f LabelFilterType) String() string

type LabelFilterer 

type LabelFilterer interface {
	Stage
	fmt.Stringer
	// contains filtered or unexported methods
}

LabelFilterer can filter extracted labels.

func NewStringLabelFilter 

func NewStringLabelFilter(m *labels.Matcher) LabelFilterer

NewStringLabelFilter creates a new label filterer which compares string label. This is the only LabelFilterer that can filter out the __error__ label. Unlike other LabelFilterer which apply conversion, if the label name doesn't exist it is compared with an empty value.

func ReduceAndLabelFilter 

func ReduceAndLabelFilter(filters []LabelFilterer) LabelFilterer

ReduceAndLabelFilter Reduces multiple label filterer into one using binary and operation.

type LabelFmt 

type LabelFmt struct {
	Name  string
	Value string

	Rename bool
}

LabelFmt is a configuration struct for formatting a label.

func NewRenameLabelFmt 

func NewRenameLabelFmt(dst, target string) LabelFmt

NewRenameLabelFmt creates a configuration to rename a label.

func NewTemplateLabelFmt 

func NewTemplateLabelFmt(dst, template string) LabelFmt

NewTemplateLabelFmt creates a configuration to format a label using text template.

type LabelsBuilder 

type LabelsBuilder struct {
	*BaseLabelsBuilder
	// contains filtered or unexported fields
}

LabelsBuilder is the same as labels.Builder but tailored for this package.

func (*LabelsBuilder) Add 

func (b *LabelsBuilder) Add(category LabelCategory, labels ...labels.Label) *LabelsBuilder

Add the labels to the builder. If a label with the same name already exists in the base labels, a suffix is added to the name.

func (*LabelsBuilder) BaseHas 

func (b *LabelsBuilder) BaseHas(key string) bool

BaseHas returns the base labels have the given key

func (*LabelsBuilder) Del 

func (b *LabelsBuilder) Del(ns ...string) *LabelsBuilder

Del deletes the label of the given name.

func (*LabelsBuilder) Get 

func (b *LabelsBuilder) Get(key string) (string, bool)

func (*LabelsBuilder) GetErr 

func (b *LabelsBuilder) GetErr() string

GetErr return the current error label value.

func (*LabelsBuilder) GetErrorDetails 

func (b *LabelsBuilder) GetErrorDetails() string

func (*LabelsBuilder) GetWithCategory 

func (b *LabelsBuilder) GetWithCategory(key string) (string, LabelCategory, bool)

GetWithCategory returns the value and the category of a labels key if it exists.

func (*LabelsBuilder) GroupedLabels 

func (b *LabelsBuilder) GroupedLabels() LabelsResult

GroupedLabels returns the LabelsResult from the builder. Groups are applied and the cache is used when possible.

func (*LabelsBuilder) HasErr 

func (b *LabelsBuilder) HasErr() bool

HasErr tells if the error label has been set.

func (*LabelsBuilder) HasErrorDetails 

func (b *LabelsBuilder) HasErrorDetails() bool

func (LabelsBuilder) Hash 

func (h LabelsBuilder) Hash(lbs labels.Labels) uint64

Hash hashes the labels

func (*LabelsBuilder) IntoMap 

func (b *LabelsBuilder) IntoMap(m map[string]string)

puts labels entries into an existing map, it is up to the caller to properly clear the map if it is going to be reused

func (*LabelsBuilder) LabelsResult 

func (b *LabelsBuilder) LabelsResult() LabelsResult

LabelsResult returns the LabelsResult from the builder. No grouping is applied and the cache is used when possible.

func (*LabelsBuilder) Map 

func (b *LabelsBuilder) Map() (map[string]string, bool)

func (*LabelsBuilder) ResetError 

func (b *LabelsBuilder) ResetError() *LabelsBuilder

func (*LabelsBuilder) ResetErrorDetails 

func (b *LabelsBuilder) ResetErrorDetails() *LabelsBuilder

func (*LabelsBuilder) Set 

func (b *LabelsBuilder) Set(category LabelCategory, n, v string) *LabelsBuilder

Set the name/value pair as a label.

func (*LabelsBuilder) SetErr 

func (b *LabelsBuilder) SetErr(err string) *LabelsBuilder

SetErr sets the error label.

func (*LabelsBuilder) SetErrorDetails 

func (b *LabelsBuilder) SetErrorDetails(desc string) *LabelsBuilder

func (*LabelsBuilder) UnsortedLabels 

func (b *LabelsBuilder) UnsortedLabels(buf labels.Labels, categories ...LabelCategory) labels.Labels

type LabelsFormatter 

type LabelsFormatter struct {
	// contains filtered or unexported fields
}

func NewLabelsFormatter 

func NewLabelsFormatter(fmts []LabelFmt) (*LabelsFormatter, error)

NewLabelsFormatter creates a new formatter that can format multiple labels at once. Either by renaming or using text template. It is not allowed to reformat the same label twice within the same formatter.

func (*LabelsFormatter) Process 

func (lf *LabelsFormatter) Process(ts int64, l []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LabelsFormatter) RequiredLabelNames 

func (lf *LabelsFormatter) RequiredLabelNames() []string

type LabelsResult 

type LabelsResult interface {
	String() string
	Labels() labels.Labels
	Stream() labels.Labels
	StructuredMetadata() labels.Labels
	Parsed() labels.Labels
	Hash() uint64
}

LabelsResult is a computed labels result that contains the labels set with associated string and hash. The is mainly used for caching and returning labels computations out of pipelines and stages.

func NewLabelsResult 

func NewLabelsResult(allLabelsStr string, hash uint64, stream, structuredMetadata, parsed labels.Labels) LabelsResult

NewLabelsResult creates a new LabelsResult. It takes the string representation of the labels, the hash of the labels and the labels categorized.

type LineExtractor 

type LineExtractor func([]byte) float64

LineExtractor extracts a float64 from a log line. 

var (
	CountExtractor LineExtractor = func(line []byte) float64 { return 1. }
	BytesExtractor LineExtractor = func(line []byte) float64 { return float64(len(line)) }
)

type LineFilterLabelFilter 

type LineFilterLabelFilter struct {
	*labels.Matcher
	Filter Filterer
}

LineFilterLabelFilter filters the desired label using an optimized line filter

func (*LineFilterLabelFilter) Process 

func (s *LineFilterLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LineFilterLabelFilter) RequiredLabelNames 

func (s *LineFilterLabelFilter) RequiredLabelNames() []string

func (*LineFilterLabelFilter) String 

func (s *LineFilterLabelFilter) String() string

overrides the matcher.String() function in case there is a regexpFilter

type LineFormatter 

type LineFormatter struct {
	*template.Template
	// contains filtered or unexported fields
}

func NewFormatter 

func NewFormatter(tmpl string) (*LineFormatter, error)

NewFormatter creates a new log line formatter from a given text template.

func (*LineFormatter) Process 

func (lf *LineFormatter) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LineFormatter) RequiredLabelNames 

func (lf *LineFormatter) RequiredLabelNames() []string

type LineMatchType 

type LineMatchType int

LineMatchType is an enum for line matching types. 

const (
	LineMatchEqual LineMatchType = iota
	LineMatchNotEqual
	LineMatchRegexp
	LineMatchNotRegexp
	LineMatchPattern
	LineMatchNotPattern
)

Possible LineMatchTypes.

func (LineMatchType) String 

func (t LineMatchType) String() string

type LogfmtExpressionParser 

type LogfmtExpressionParser struct {
	// contains filtered or unexported fields
}

func NewLogfmtExpressionParser 

func NewLogfmtExpressionParser(expressions []LabelExtractionExpr, strict bool) (*LogfmtExpressionParser, error)

func (*LogfmtExpressionParser) Process 

func (l *LogfmtExpressionParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LogfmtExpressionParser) RequiredLabelNames 

func (l *LogfmtExpressionParser) RequiredLabelNames() []string

type LogfmtParser 

type LogfmtParser struct {
	// contains filtered or unexported fields
}

func NewLogfmtParser 

func NewLogfmtParser(strict, keepEmpty bool) *LogfmtParser

NewLogfmtParser creates a parser that can extract labels from a logfmt log line. Each keyval is extracted into a respective label.

func (*LogfmtParser) Process 

func (l *LogfmtParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*LogfmtParser) RequiredLabelNames 

func (l *LogfmtParser) RequiredLabelNames() []string

type Matcher 

type Matcher interface {
	Matches(test Checker) bool
}

Matcher is a interface to match log lines against a Checker. This works in the opposite direction of Filterer. Whereas Filterer.Filter checks if an input log line satisfies the filter, Matcher.Matches checks if a filter satisfies an input log line (or regexp).

type MatcherFilterer 

type MatcherFilterer interface {
	Matcher
	Filterer
}

func ChainOrMatcherFilterer 

func ChainOrMatcherFilterer(curr, new MatcherFilterer) MatcherFilterer

ChainOrMatcherFilterer is a syntax sugar to chain multiple `or` filters. (1 or many)

func NewAndFilter 

func NewAndFilter(left MatcherFilterer, right MatcherFilterer) MatcherFilterer

NewAndFilter creates a new filter which matches only if left and right matches.

func NewNotFilter 

func NewNotFilter(base MatcherFilterer) MatcherFilterer

NewNotFilter creates a new filter which matches only if the base filter doesn't match. If the base filter is a `or` it will recursively simplify with `and` operations.

func WrapFilterer 

func WrapFilterer(f Filterer) MatcherFilterer

func WrapMatcher 

func WrapMatcher(m Matcher) MatcherFilterer

type NewMatcherFiltererFunc 

type NewMatcherFiltererFunc func(match []byte, caseInsensitive bool) MatcherFilterer

type NoopLabelFilter 

type NoopLabelFilter struct {
	*labels.Matcher
}

func (NoopLabelFilter) Process 

func (NoopLabelFilter) Process(_ int64, line []byte, _ *LabelsBuilder) ([]byte, bool)

func (NoopLabelFilter) RequiredLabelNames 

func (NoopLabelFilter) RequiredLabelNames() []string

func (NoopLabelFilter) String 

func (f NoopLabelFilter) String() string

type NumericLabelFilter 

type NumericLabelFilter struct {
	Name  string
	Value float64
	Type  LabelFilterType
	// contains filtered or unexported fields
}

func NewNumericLabelFilter 

func NewNumericLabelFilter(t LabelFilterType, name string, v float64) *NumericLabelFilter

NewNumericLabelFilter creates a new label filterer which parses float64 string representation (5.2) from the value of the named label and compares it with the given f value.

func (*NumericLabelFilter) Process 

func (n *NumericLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*NumericLabelFilter) RequiredLabelNames 

func (n *NumericLabelFilter) RequiredLabelNames() []string

func (*NumericLabelFilter) String 

func (n *NumericLabelFilter) String() string

type ParserHint 

type ParserHint interface {
	// Tells if a label with the given key should be extracted.
	ShouldExtract(key string) bool

	// Tells if there's any hint that start with the given prefix.
	// This allows to speed up key searching in nested structured like json.
	ShouldExtractPrefix(prefix string) bool

	// Tells if we should not extract any labels.
	// For example in :
	//		 sum(rate({app="foo"} | json [5m]))
	// We don't need to extract any labels from the log line.
	NoLabels() bool

	// Holds state about what's already been extracted for the associated
	// labels. This assumes that only required labels are ever extracted
	RecordExtracted(string)

	// Returns true if all the required labels have already been extracted
	AllRequiredExtracted() bool

	// Resets the state of extracted labels
	Reset()

	// PreserveError returns true when parsing errors were specifically requested
	PreserveError() bool

	// ShouldContinueParsingLine returns true when there is no label matcher for the
	// provided label or the passed label and value match what's in the pipeline
	ShouldContinueParsingLine(labelName string, lbs *LabelsBuilder) bool
}

ParserHint are hints given to LogQL parsers. This is specially useful for parser that extract implicitly all possible label keys. This is used only within metric queries since it's rare that you need all label keys. For example in the following expression: 

sum by (status_code) (rate({app="foo"} | json [5m]))

All we need to extract is the status_code in the json parser.

func NoParserHints 

func NoParserHints() ParserHint

type PatternParser 

type PatternParser struct {
	// contains filtered or unexported fields
}

func NewPatternParser 

func NewPatternParser(pn string) (*PatternParser, error)

func (*PatternParser) Process 

func (l *PatternParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*PatternParser) RequiredLabelNames 

func (l *PatternParser) RequiredLabelNames() []string

type Pipeline 

type Pipeline interface {
	ForStream(labels labels.Labels) StreamPipeline
	Reset()
}

Pipeline can create pipelines for each log stream.

func NewFilteringPipeline 

func NewFilteringPipeline(f []PipelineFilter, p Pipeline) Pipeline

NewFilteringPipeline creates a pipeline where entries from the underlying log stream are filtered by pipeline filters before being passed to the pipeline representing the queried data. Filters are always upstream of the pipeline

func NewNoopPipeline 

func NewNoopPipeline() Pipeline

NewNoopPipeline creates a pipelines that does not process anything and returns log streams as is.

func NewPipeline 

func NewPipeline(stages []Stage) Pipeline

NewPipeline creates a new pipeline for a given set of stages.

type PipelineFilter 

type PipelineFilter struct {
	Start    int64
	End      int64
	Matchers []*labels.Matcher
	Pipeline Pipeline
}

PipelineFilter contains a set of matchers and a pipeline that, when matched, causes an entry from a log stream to be skipped. Matching entries must also fall between 'start' and 'end', inclusive

type PipelineWrapper 

type PipelineWrapper interface {
	Wrap(ctx context.Context, pipeline Pipeline, query, tenant string) Pipeline
}

PipelineWrapper takes a pipeline, wraps it is some desired functionality and returns a new pipeline

type RegexSimplifier 

type RegexSimplifier struct {
	// contains filtered or unexported fields
}

func NewRegexSimplifier 

func NewRegexSimplifier(
	newContainsFilter NewMatcherFiltererFunc,
	newEqualFilter NewMatcherFiltererFunc,
) *RegexSimplifier

func (*RegexSimplifier) Simplify 

func (s *RegexSimplifier) Simplify(reg *syntax.Regexp, isLabel bool) (MatcherFilterer, bool)

Simplify a regexp expression by replacing it, when possible, with a succession of literal filters. For example `(foo|bar)` will be replaced by `containsFilter(foo) or containsFilter(bar)`

type RegexpParser 

type RegexpParser struct {
	// contains filtered or unexported fields
}

func NewRegexpParser 

func NewRegexpParser(re string) (*RegexpParser, error)

NewRegexpParser creates a new log stage that can extract labels from a log line using a regex expression. The regex expression must contains at least one named match. If the regex doesn't match the line is not filtered out.

func (*RegexpParser) Process 

func (r *RegexpParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*RegexpParser) RequiredLabelNames 

func (r *RegexpParser) RequiredLabelNames() []string

type SampleExtractor 

type SampleExtractor interface {
	ForStream(labels labels.Labels) StreamSampleExtractor
}

SampleExtractor creates StreamSampleExtractor that can extract samples for a given log stream.

func LabelExtractorWithStages 

func LabelExtractorWithStages(
	labelName, conversion string,
	groups []string, without, noLabels bool,
	preStages []Stage,
	postFilter Stage,
) (SampleExtractor, error)

LabelExtractorWithStages creates a SampleExtractor that will extract metrics from a labels. A set of log stage is executed before the conversion. A Filtering stage is executed after the conversion allowing to remove sample containing the __error__ label.

func NewFilteringSampleExtractor 

func NewFilteringSampleExtractor(f []PipelineFilter, e SampleExtractor) SampleExtractor

NewFilteringSampleExtractor creates a sample extractor where entries from the underlying log stream are filtered by pipeline filters before being passed to extract samples. Filters are always upstream of the extractor.

func NewLineSampleExtractor 

func NewLineSampleExtractor(ex LineExtractor, stages []Stage, groups []string, without, noLabels bool) (SampleExtractor, error)

NewLineSampleExtractor creates a SampleExtractor from a LineExtractor. Multiple log stages are run before converting the log line.

type SampleExtractorWrapper 

type SampleExtractorWrapper interface {
	Wrap(ctx context.Context, extractor SampleExtractor, query, tenant string) SampleExtractor
}

SampleExtractorWrapper takes an extractor, wraps it is some desired functionality and returns a new pipeline

type Simplifier 

type Simplifier interface {
	Simplify(reg *syntax.Regexp, isLabel bool) (Filterer, bool)
}

type Stage 

type Stage interface {
	Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)
	RequiredLabelNames() []string
}

Stage is a single step of a Pipeline. A Stage implementation should never mutate the line passed, but instead either return the line unchanged or allocate a new line. 

var NoopStage Stage = &noopStage{}

NoopStage is a stage that doesn't process a log line.

func ReduceStages 

func ReduceStages(stages []Stage) Stage

ReduceStages reduces multiple stages into one.

type StageFunc 

type StageFunc struct {
	// contains filtered or unexported fields
}

func (StageFunc) Process 

func (fn StageFunc) Process(ts int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (StageFunc) RequiredLabelNames 

func (fn StageFunc) RequiredLabelNames() []string

type StreamPipeline 

type StreamPipeline interface {
	BaseLabels() LabelsResult
	// Process processes a log line and returns the transformed line and the labels.
	// The buffer returned for the log line can be reused on subsequent calls to Process and therefore must be copied.
	Process(ts int64, line []byte, structuredMetadata ...labels.Label) (resultLine []byte, resultLabels LabelsResult, matches bool)
	ProcessString(ts int64, line string, structuredMetadata ...labels.Label) (resultLine string, resultLabels LabelsResult, matches bool)
	ReferencedStructuredMetadata() bool
}

StreamPipeline transform and filter log lines and labels. A StreamPipeline never mutate the received line.

func NewStreamPipeline 

func NewStreamPipeline(stages []Stage, labelsBuilder *LabelsBuilder) StreamPipeline

type StreamSampleExtractor 

type StreamSampleExtractor interface {
	BaseLabels() LabelsResult
	Process(ts int64, line []byte, structuredMetadata ...labels.Label) (float64, LabelsResult, bool)
	ProcessString(ts int64, line string, structuredMetadata ...labels.Label) (float64, LabelsResult, bool)
	ReferencedStructuredMetadata() bool
}

StreamSampleExtractor extracts sample for a log line. A StreamSampleExtractor never mutate the received line.

type StringLabelFilter 

type StringLabelFilter struct {
	*labels.Matcher
}

func (*StringLabelFilter) Process 

func (s *StringLabelFilter) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (*StringLabelFilter) RequiredLabelNames 

func (s *StringLabelFilter) RequiredLabelNames() []string

type UnpackParser 

type UnpackParser struct {
	// contains filtered or unexported fields
}

func NewUnpackParser 

func NewUnpackParser() *UnpackParser

NewUnpackParser creates a new unpack stage. The unpack stage will parse a json log line as map[string]string where each key will be translated into labels. A special key _entry will also be used to replace the original log line. This is to be used in conjunction with Promtail pack stage. see https://grafana.com/docs/loki/latest/clients/promtail/stages/pack/

func (*UnpackParser) Process 

func (u *UnpackParser) Process(_ int64, line []byte, lbs *LabelsBuilder) ([]byte, bool)

func (UnpackParser) RequiredLabelNames 

func (UnpackParser) RequiredLabelNames() []string

Source Files

View all Source files

Directories

Path Synopsis
Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.
 Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.