tql

README

Telemetry Query Language

The Telemetry Query Language is a query language for transforming open telemetry data based on the OpenTelemetry Collector Processing Exploration.

This package reads in TQL queries and converts them to invokable Booleans and functions based on the TQL's grammar.

The TQL is signal agnostic; it is not aware of the type of telemetry on which it will operate. Instead, the Booleans and functions returned by the package must be passed a TransformContext, which provide access to the signal's telemetry. Telemetry data can be accessed and updated through Getters and Setters.

Grammar

The TQL grammar includes Invocations, Values and Expressions.

Invocations

Invocations represent a function call. Invocations are made up of 2 parts:

• a string identifier. The string identifier must start with a letter or an underscore (_).
• zero or more Values (comma separated) surrounded by parentheses (()).

The TQL does not define any function implementations. Users must supply a map between string identifiers and the actual function implementation. The TQL will use this map and reflection to generate Invocations, that can then be invoked by the user.

Example Invocations

• drop()
• set(field, 1)

Invocation parameters

The TQL will use reflection to determine parameter types when parsing an invocation within a statement. When interpreting slice parameter types, the TQL will attempt to build the slice from all remaining Values in the Invocation's arguments. As a result, function implementations of Invocations may only contain one slice argument and it must be the last argument in the function definition. See function syntax guidelines for more details.

The following types are supported for single parameter values:

• Setter
• GetSetter
• Getter
• Enum
• string
• float64
• int64
• bool

For slice parameters, the following types are supported:

• string
• float64
• int64
• uint8. Slices of bytes will be interpreted as a byte array.
• Getter

Values

Values are passed as input to an Invocation or are used in an Expression. Values can take the form of:

• Paths.
• Literals.
• Enums.
• Invocations.

Invocations as Values allows calling functions as parameters to other functions. See Invocations for details on Invocation syntax.

Paths

A Path Value is a reference to a telemetry field. Paths are made up of lowercase identifiers, dots (.), and square brackets combined with a string key (["key"]). The interpretation of a Path is NOT implemented by the TQL. Instead, the user must provide a PathExpressionParser that the TQL can use to interpret paths. As a result, how the Path parts are used is up to the user. However, it is recommended, that the parts be used like so:

• Identifiers are used to map to a telemetry field.
• Dots (.) are used to separate nested fields.
• Square brackets and keys (["key"]) are used to access maps or slices.

Example Paths

• name
• value_double
• resource.name
• resource.attributes["key"]

Literals

Literals are literal interpretations of the Value into a Go value. Accepted literals are:

• Strings. Strings are represented as literals by surrounding the string in double quotes ("").
• Ints. Ints are represented by any digit, optionally prepended by plus (+) or minus (-). Internally the TQL represents all ints as int64
• Floats. Floats are represented by digits separated by a dot (.), optionally prepended by plus (+) or minus (-). The leading digit is optional. Internally the TQL represents all Floats as float64.
• Bools. Bools are represented by the exact strings true and false.
• Nil. Nil is represented by the exact string nil.
• Byte slices. Byte slices are represented via a hex string prefaced with 0x

Example Literals

• "a string"
• 1, -1
• 1.5, -.5
• true, false
• nil,
• 0x0001

Enums

Enums are uppercase identifiers that get interpreted during parsing and converted to an int64. The interpretation of an Enum is NOT implemented by the TQL. Instead, the user must provide a EnumParser that the TQL can use to interpret the Enum. The EnumParser returns an int64 instead of a function, which means that the Enum's numeric value is retrieved during parsing instead of during execution.

Within the grammar Enums are always used as int64. As a result, the Enum's symbol can be used as if it is an Int value.

When defining a function that will be used as an Invocation by the TQL, if the function needs to take an Enum then the function must use the Enum type for that argument, not an int64.

Expressions

Expressions allow a decision to be made about whether an Invocation should be called. Expressions are optional. When used, the parsed query will include a Condition, which can be used to evaluate the result of the query's Expression. Expressions always evaluate to a boolean value (true or false).

Expressions consist of the literal string where followed by one or more Booleans (see below). Booleans can be joined with the literal strings and and or. Note that and expressions have higher precedence than or. Expressions can be grouped with parentheses to override evaluation precedence.

Booleans

Booleans can be either:

• A literal boolean value (true or false).
• A Comparison, made up of a left Value, an operator, and a right Value. See Values for details on what a Value can be.

Operators determine how the two Values are compared.

The valid operators are:

• Equal (==). Tests if the left and right Values are equal (see the Comparison Rules below).
• Not Equal (!=). Tests if the left and right Values are not equal.
• Less Than (<). Tests if left is less than right.
• Greater Than (>). Tests if left is greater than right.
• Less Than or Equal To (<=). Tests if left is less than or equal to right.
• Greater Than or Equal to (>=). Tests if left is greater than or equal to right.

Comparison Rules

The table below describes what happens when two Values are compared. Value types are provided by the user of TQL. All of the value types supported by TQL are listed in this table.

If numeric values are of different types, they are compared as float64.

For numeric values and strings, the comparison rules are those implemented by Go. Numeric values are done with signed comparisons. For binary values, false is considered to be less than true.

For values that are not one of the basic primitive types, the only valid comparisons are Equal and Not Equal, which are implemented using Go's standard == and != operators.

A not equal notation in the table below means that the "!=" operator returns true, but any other operator returns false. Note that a nil byte array is considered equivalent to nil.

base type	bool	int64	float64	string	Bytes	nil
bool	normal, T>F	not equal	not equal	not equal	not equal	not equal
int64	not equal	compared as largest	compared as float64	not equal	not equal	not equal
float64	not equal	compared as float64	compared as largest	not equal	not equal	not equal
string	not equal	not equal	not equal	normal (compared as Go strings)	not equal	not equal
Bytes	not equal	not equal	not equal	not equal	byte-for-byte comparison	[]byte(nil) == nil
nil	not equal	not equal	not equal	not equal	[]byte(nil) == nil	true for equality only

Accessing signal telemetry

Access to signal telemetry is provided to TQL functions through a TransformContext that is created by the user and passed during statement evaluation. To allow functions to operate on the TransformContext, the TQL provides Getter, Setter, and GetSetter interfaces.

Getters and Setters

Getters allow for reading the following types of data. See the respective section of each Value type for how they are interpreted.

• Paths.
• Enums.
• Literals.
• Invocations.

It is possible to update the Value in a telemetry field using a Setter. For read and write access, the GetSetter interface extends both interfaces.

Logging inside a TQL function

To emit logs inside a TQL function, add a parameter of type Logger to the function signature. The TQL will then inject a logger instance provided by the component that can be used to emit logs.

Examples

These examples contain a SQL-like declarative language. Applied statements interact with only one signal, but statements can be declared across multiple signals. Functions used in examples are indicative of what could be useful, but are not implemented by the TQL itself.

Remove a forbidden attribute

traces:
  delete(attributes["http.request.header.authorization"])
metrics:
  delete(attributes["http.request.header.authorization"])
logs:
  delete(attributes["http.request.header.authorization"])

Remove all attributes except for some

traces:
  keep_keys(attributes, "http.method", "http.status_code")
metrics:
  keep_keys(attributes, "http.method", "http.status_code")
logs:
  keep_keys(attributes, "http.method", "http.status_code")

Reduce cardinality of an attribute

traces:
  replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}")

Reduce cardinality of a span name

traces:
  replace_match(name, "GET /user/*/list/*", "GET /user/{userId}/list/{listId}")

Reduce cardinality of any matching attribute

traces:
  replace_all_matches(attributes, "/user/*/list/*", "/user/{userId}/list/{listId}")

Decrease the size of the telemetry payload

traces:
  delete(resource.attributes["process.command_line"])
metrics:
  delete(resource.attributes["process.command_line"])
logs:
  delete(resource.attributes["process.command_line"])

Drop specific telemetry

metrics:
  drop() where attributes["http.target"] == "/health"

Attach information from resource into telemetry

metrics:
  set(attributes["k8s_pod"], resource.attributes["k8s.pod.name"])

Decorate error spans with additional information

traces:
  set(attributes["whose_fault"], "theirs") where attributes["http.status"] == 400 or attributes["http.status"] == 404
  set(attributes["whose_fault"], "ours") where attributes["http.status"] == 500

Group spans by trace ID

traces:
  group_by(trace_id, 2m)

Update a spans ID

logs:
  set(span_id, SpanID(0x0000000000000000))
traces:
  set(span_id, SpanID(0x0000000000000000))

Create utilization metric from base metrics.

metrics:
  create_gauge("pod.cpu.utilized", read_gauge("pod.cpu.usage") / read_gauge("node.cpu.limit")

Documentation

Index

• type BoolExpressionEvaluator
• type Boolean
  • func (b *Boolean) Capture(values []string) error
• type BooleanExpression
• type BooleanValue
• type Bytes
  • func (b *Bytes) Capture(values []string) error
• type CompareOp
  • func (c *CompareOp) Capture(values []string) error
  • func (c CompareOp) String() string
• type Comparison
• type Enum
• type EnumParser
• type EnumSymbol
• type ExprFunc
• type Field
• type GetSetter
• type Getter
• type Invocation
• type IsNil
  • func (n *IsNil) Capture(_ []string) error
• type Literal
  • func (l Literal) Get(ctx TransformContext) interface{}
• type Logger
• type NoOpLogger
  • func (nol NoOpLogger) Error(msg string)
  • func (nol NoOpLogger) Info(msg string)
  • func (nol NoOpLogger) WithFields(field map[string]any) Logger
• type OpAndBooleanValue
• type OpOrTerm
• type ParsedQuery
• type Parser
  • func NewParser(functions map[string]interface{}, pathParser PathExpressionParser, ...) Parser
  • func (p *Parser) NewFunctionCall(inv Invocation) (ExprFunc, error)
  • func (p *Parser) NewGetter(val Value) (Getter, error)
  • func (p *Parser) ParseQueries(statements []string) ([]Query, error)
• type Path
• type PathExpressionParser
• type Query
• type Setter
• type StandardGetSetter
  • func (path StandardGetSetter) Get(ctx TransformContext) interface{}
  • func (path StandardGetSetter) Set(ctx TransformContext, val interface{})
• type Term
• type TransformContext
• type Value

Types

type BoolExpressionEvaluator

type BoolExpressionEvaluator = func(ctx TransformContext) bool

BoolExpressionEvaluator is a function that returns the result.

type Boolean

type Boolean bool

Boolean Type for capturing booleans, see: https://github.com/alecthomas/participle#capturing-boolean-value

func (*Boolean) Capture

func (b *Boolean) Capture(values []string) error

type BooleanExpression

type BooleanExpression struct {
	Left  *Term       `parser:"@@"`
	Right []*OpOrTerm `parser:"@@*"`
}

BooleanExpression represents a true/false decision expressed as an arbitrary number of terms separated by OR.

type BooleanValue

type BooleanValue struct {
	Comparison *Comparison        `parser:"( @@"`
	ConstExpr  *Boolean           `parser:"| @Boolean"`
	SubExpr    *BooleanExpression `parser:"| '(' @@ ')' )"`
}

BooleanValue represents something that evaluates to a boolean -- either an equality or inequality, explicit true or false, or a parenthesized subexpression.

type Bytes

type Bytes []byte

Bytes type for capturing byte arrays

func (*Bytes) Capture

func (b *Bytes) Capture(values []string) error

type CompareOp

type CompareOp int

CompareOp is the type of a comparison operator.

const (
	EQ CompareOp = iota
	NE
	LT
	LTE
	GTE
	GT
)

These are the allowed values of a CompareOp

func (*CompareOp) Capture

func (c *CompareOp) Capture(values []string) error

Capture is how the parser converts an operator string to a CompareOp.

func (CompareOp) String

func (c CompareOp) String() string

String() for CompareOp gives us more legible test results and error messages.

type Comparison

type Comparison struct {
	Left  Value     `parser:"@@"`
	Op    CompareOp `parser:"@OpComparison"`
	Right Value     `parser:"@@"`
}

Comparison represents an optional boolean condition.

type Enum

type Enum int64

type EnumParser

type EnumParser func(*EnumSymbol) (*Enum, error)

type EnumSymbol

type EnumSymbol string

type ExprFunc

type ExprFunc func(ctx TransformContext) interface{}

type Field

type Field struct {
	Name   string  `parser:"@Lowercase"`
	MapKey *string `parser:"( '[' @String ']' )?"`
}

Field is an item within a Path.

type GetSetter

type GetSetter interface {
	Getter
	Setter
}

type Getter

type Getter interface {
	Get(ctx TransformContext) interface{}
}

type Invocation

type Invocation struct {
	Function  string  `parser:"@(Uppercase | Lowercase)+"`
	Arguments []Value `parser:"'(' ( @@ ( ',' @@ )* )? ')'"`
}

Invocation represents a function call.

type IsNil

type IsNil bool

func (*IsNil) Capture

func (n *IsNil) Capture(_ []string) error

type Literal

type Literal struct {
	Value interface{}
}

func (Literal) Get

func (l Literal) Get(ctx TransformContext) interface{}

type Logger

type Logger interface {
	WithFields(field map[string]any) Logger
	Info(msg string)
	Error(msg string)
}

Logger allows printing logs inside TQL functions using a logging framework provided by the component using the TQL.

type NoOpLogger

type NoOpLogger struct{}

func (NoOpLogger) Error

func (nol NoOpLogger) Error(msg string)

func (NoOpLogger) Info

func (nol NoOpLogger) Info(msg string)

func (NoOpLogger) WithFields

func (nol NoOpLogger) WithFields(field map[string]any) Logger

type OpAndBooleanValue

type OpAndBooleanValue struct {
	Operator string        `parser:"@OpAnd"`
	Value    *BooleanValue `parser:"@@"`
}

OpAndBooleanValue represents the right side of an AND boolean expression.

type OpOrTerm

type OpOrTerm struct {
	Operator string `parser:"@OpOr"`
	Term     *Term  `parser:"@@"`
}

OpOrTerm represents the right side of an OR boolean expression.

type ParsedQuery

type ParsedQuery struct {
	Invocation  Invocation         `parser:"@@"`
	WhereClause *BooleanExpression `parser:"( 'where' @@ )?"`
}

ParsedQuery represents a parsed query. It is the entry point into the query DSL.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

func NewParser

func NewParser(functions map[string]interface{}, pathParser PathExpressionParser, enumParser EnumParser, logger Logger) Parser

func (*Parser) NewFunctionCall

func (p *Parser) NewFunctionCall(inv Invocation) (ExprFunc, error)

NewFunctionCall Visible for testing

func (*Parser) NewGetter

func (p *Parser) NewGetter(val Value) (Getter, error)

func (*Parser) ParseQueries

func (p *Parser) ParseQueries(statements []string) ([]Query, error)

type Path

type Path struct {
	Fields []Field `parser:"@@ ( '.' @@ )*"`
}

Path represents a telemetry path expression.

type PathExpressionParser

type PathExpressionParser func(*Path) (GetSetter, error)

type Query

type Query struct {
	Function  ExprFunc
	Condition BoolExpressionEvaluator
}

Query holds a top level Query for processing telemetry data. A Query is a combination of a function invocation and the expression to match telemetry for invoking the function.

type Setter

type Setter interface {
	Set(ctx TransformContext, val interface{})
}

type StandardGetSetter

type StandardGetSetter struct {
	Getter func(ctx TransformContext) interface{}
	Setter func(ctx TransformContext, val interface{})
}

func (StandardGetSetter) Get

func (path StandardGetSetter) Get(ctx TransformContext) interface{}

func (StandardGetSetter) Set

func (path StandardGetSetter) Set(ctx TransformContext, val interface{})

type Term

type Term struct {
	Left  *BooleanValue        `parser:"@@"`
	Right []*OpAndBooleanValue `parser:"@@*"`
}

Term represents an arbitrary number of boolean values joined by AND.

type TransformContext

type TransformContext interface {
	GetItem() interface{}
	GetInstrumentationScope() pcommon.InstrumentationScope
	GetResource() pcommon.Resource
}

type Value

type Value struct {
	Invocation *Invocation `parser:"( @@"`
	Bytes      *Bytes      `parser:"| @Bytes"`
	String     *string     `parser:"| @String"`
	Float      *float64    `parser:"| @Float"`
	Int        *int64      `parser:"| @Int"`
	Bool       *Boolean    `parser:"| @Boolean"`
	IsNil      *IsNil      `parser:"| @'nil'"`
	Enum       *EnumSymbol `parser:"| @Uppercase"`
	Path       *Path       `parser:"| @@ )"`
}

Value represents a part of a parsed query which is resolved to a value of some sort. This can be a telemetry path expression, function call, or literal.