metricsview

README

runtime/metricsview/

This package contains the core logic for Rill's metrics layer (semantic layer). Namely, that includes validating metrics views and generating SQL queries for querying dimensions and measures defined in a metrics view.

Some other code files relevant to metrics views that are not found in this package:

• proto/rill/runtime/v1/resources.proto: defines the spec for a metrics view
• runtime/compilers/rillv1/parse_metrics_view.go: contains the logic for parsing a metrics view spec from a YAML file

Documentation

Index

• func ExpressionToString(e *Expression) (string, error)
• type AST
  • func NewAST(mv *runtimev1.MetricsViewSpec, sec *runtime.ResolvedSecurity, qry *Query, ...) (*AST, error)
  • func (a *AST) SQL() (string, []any, error)
• type Condition
• type Dimension
• type DimensionCompute
• type DimensionComputeTimeFloor
• type Executor
  • func NewExecutor(ctx context.Context, rt *runtime.Runtime, instanceID string, ...) (*Executor, error)
  • func (e *Executor) Cacheable(qry *Query) bool
  • func (e *Executor) Close()
  • func (e *Executor) Export(ctx context.Context, qry *Query, executionTime *time.Time, ...) (string, error)
  • func (e *Executor) Query(ctx context.Context, qry *Query, executionTime *time.Time) (*drivers.Result, error)
  • func (e *Executor) Schema(ctx context.Context) (*runtimev1.StructType, error)
  • func (e *Executor) ValidateMetricsView(ctx context.Context) error
  • func (e *Executor) ValidateQuery(qry *Query) error
  • func (e *Executor) Watermark(ctx context.Context) (time.Time, error)
• type ExprNode
• type Expression
  • func NewExpressionFromProto(expr *runtimev1.Expression) *Expression
• type FieldNode
• type JoinType
• type Measure
• type MeasureCompute
  • func (m *MeasureCompute) Validate() error
• type MeasureComputeComparisonDelta
• type MeasureComputeComparisonRatio
• type MeasureComputeComparisonValue
• type MeasureComputeCountDistinct
• type Operator
  • func (o Operator) Valid() bool
• type OrderFieldNode
• type Query
• type SelectNode
• type Sort
• type Spine
• type Subquery
• type TimeGrain
  • func TimeGrainFromProto(t runtimev1.TimeGrain) TimeGrain
  • func (t TimeGrain) ToProto() runtimev1.TimeGrain
  • func (t TimeGrain) ToTimeutil() timeutil.TimeGrain
  • func (t TimeGrain) Valid() bool
• type TimeRange
  • func (tr *TimeRange) IsZero() bool
• type TimeSpine
• type WhereSpine

Functions

func ExpressionToString

func ExpressionToString(e *Expression) (string, error)

Types

type AST

type AST struct {
	// Root of the AST
	Root *SelectNode
	// List of CTEs to add to the query
	CTEs []*SelectNode
	// contains filtered or unexported fields
}

AST is the abstract syntax tree for a metrics SQL query.

func NewAST

func NewAST(mv *runtimev1.MetricsViewSpec, sec *runtime.ResolvedSecurity, qry *Query, dialect drivers.Dialect) (*AST, error)

NewAST builds a new SQL AST based on a metrics query.

Dynamic time ranges in the qry must be resolved to static start/end timestamps before calling this function. This is due to NewAST not being able (or intended) to resolve external time anchors such as watermarks.

The qry's PivotOn must be empty. Pivot queries must be rewritten/handled upstream of NewAST.

func (*AST) SQL

func (a *AST) SQL() (string, []any, error)

SQL builds a SQL query from the AST. It returns the query and query arguments to be passed to the database driver.

type Condition

type Condition struct {
	Operator    Operator      `mapstructure:"op"`
	Expressions []*Expression `mapstructure:"exprs"`
}

type Dimension

type Dimension struct {
	Name    string            `mapstructure:"name"`
	Compute *DimensionCompute `mapstructure:"compute"`
}

type DimensionCompute

type DimensionCompute struct {
	TimeFloor *DimensionComputeTimeFloor `mapstructure:"time_floor"`
}

type DimensionComputeTimeFloor

type DimensionComputeTimeFloor struct {
	Dimension string    `mapstructure:"dimension"`
	Grain     TimeGrain `mapstructure:"grain"`
}

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor is capable of executing queries and other operations against a metrics view.

func NewExecutor

func NewExecutor(ctx context.Context, rt *runtime.Runtime, instanceID string, mv *runtimev1.MetricsViewSpec, sec *runtime.ResolvedSecurity, priority int) (*Executor, error)

NewExecutor creates a new Executor for the provided metrics view.

func (*Executor) Cacheable

func (e *Executor) Cacheable(qry *Query) bool

Cacheable returns whether the result of running the given query is cacheable.

func (*Executor) Close

func (e *Executor) Close()

Close releases the resources held by the Executor.

func (*Executor) Export

func (e *Executor) Export(ctx context.Context, qry *Query, executionTime *time.Time, format drivers.FileFormat) (string, error)

Export executes and exports the provided query against the metrics view. It returns a path to a temporary file containing the export. The caller is responsible for cleaning up the file.

func (*Executor) Query

func (e *Executor) Query(ctx context.Context, qry *Query, executionTime *time.Time) (*drivers.Result, error)

Query executes the provided query against the metrics view.

func (*Executor) Schema

func (e *Executor) Schema(ctx context.Context) (*runtimev1.StructType, error)

Schema returns a schema for the metrics view's dimensions and measures.

func (*Executor) ValidateMetricsView

func (e *Executor) ValidateMetricsView(ctx context.Context) error

ValidateMetricsView validates the dimensions and measures in the executor's metrics view.

func (*Executor) ValidateQuery

func (e *Executor) ValidateQuery(qry *Query) error

ValidateQuery validates the provided query against the executor's metrics view.

func (*Executor) Watermark

func (e *Executor) Watermark(ctx context.Context) (time.Time, error)

Watermark returns the current watermark of the metrics view. If the watermark resolves to null, it defaults to the current time.

type ExprNode

type ExprNode struct {
	Expr string
	Args []any
}

ExprNode represents an expression for a WHERE clause.

type Expression

type Expression struct {
	Name      string     `mapstructure:"name"`
	Value     any        `mapstructure:"val"`
	Condition *Condition `mapstructure:"cond"`
	Subquery  *Subquery  `mapstructure:"subquery"`
}

func NewExpressionFromProto

func NewExpressionFromProto(expr *runtimev1.Expression) *Expression

type FieldNode

type FieldNode struct {
	Name       string
	Label      string
	Expr       string
	AutoUnnest bool
}

FieldNode represents a column in a SELECT clause. It also carries metadata related to the dimension/measure it was derived from. The Name must always match a the name of a dimension/measure in the metrics view or a computed field specified in the request. This means that if two columns in different places in the AST have the same Name, they're guaranteed to resolve to the same value.

type JoinType

type JoinType string

JoinType represents types of SQL joins.

const (
	JoinTypeUnspecified JoinType = ""
	JoinTypeFull        JoinType = "FULL OUTER"
	JoinTypeLeft        JoinType = "LEFT OUTER"
	JoinTypeRight       JoinType = "RIGHT OUTER"
)

type Measure

type Measure struct {
	Name    string          `mapstructure:"name"`
	Compute *MeasureCompute `mapstructure:"compute"`
}

type MeasureCompute

type MeasureCompute struct {
	Count           bool                           `mapstructure:"count"`
	CountDistinct   *MeasureComputeCountDistinct   `mapstructure:"count_distinct"`
	ComparisonValue *MeasureComputeComparisonValue `mapstructure:"comparison_value"`
	ComparisonDelta *MeasureComputeComparisonDelta `mapstructure:"comparison_delta"`
	ComparisonRatio *MeasureComputeComparisonRatio `mapstructure:"comparison_ratio"`
}

func (*MeasureCompute) Validate

func (m *MeasureCompute) Validate() error

type MeasureComputeComparisonDelta

type MeasureComputeComparisonDelta struct {
	Measure string `mapstructure:"measure"`
}

type MeasureComputeComparisonRatio

type MeasureComputeComparisonRatio struct {
	Measure string `mapstructure:"measure"`
}

type MeasureComputeComparisonValue

type MeasureComputeComparisonValue struct {
	Measure string `mapstructure:"measure"`
}

type MeasureComputeCountDistinct

type MeasureComputeCountDistinct struct {
	Dimension string `mapstructure:"dimension"`
}

type Operator

type Operator string

const (
	OperatorUnspecified Operator = ""
	OperatorEq          Operator = "eq"
	OperatorNeq         Operator = "neq"
	OperatorLt          Operator = "lt"
	OperatorLte         Operator = "lte"
	OperatorGt          Operator = "gt"
	OperatorGte         Operator = "gte"
	OperatorIn          Operator = "in"
	OperatorNin         Operator = "nin"
	OperatorIlike       Operator = "ilike"
	OperatorNilike      Operator = "nilike"
	OperatorOr          Operator = "or"
	OperatorAnd         Operator = "and"
)

func (Operator) Valid

func (o Operator) Valid() bool

type OrderFieldNode

type OrderFieldNode struct {
	Name string
	Desc bool
}

OrderFieldNode represents a field in an ORDER BY clause.

type Query

type Query struct {
	MetricsView         string      `mapstructure:"metrics_view"`
	Dimensions          []Dimension `mapstructure:"dimensions"`
	Measures            []Measure   `mapstructure:"measures"`
	PivotOn             []string    `mapstructure:"pivot_on"`
	Spine               *Spine      `mapstructure:"spine"`
	Sort                []Sort      `mapstructure:"sort"`
	TimeRange           *TimeRange  `mapstructure:"time_range"`
	ComparisonTimeRange *TimeRange  `mapstructure:"comparison_time_range"`
	Where               *Expression `mapstructure:"where"`
	Having              *Expression `mapstructure:"having"`
	Limit               *int64      `mapstructure:"limit"`
	Offset              *int64      `mapstructure:"offset"`
	TimeZone            string      `mapstructure:"time_zone"`
	Label               bool        `mapstructure:"label"`
}

type SelectNode

type SelectNode struct {
	Alias                string           // Alias for the node used by outer SELECTs to reference it.
	IsCTE                bool             // Whether this node is a Common Table Expression
	DimFields            []FieldNode      // Dimensions fields to select
	MeasureFields        []FieldNode      // Measure fields to select
	FromTable            *string          // Underlying table expression to select from (if set, FromSelect must not be set)
	FromSelect           *SelectNode      // Sub-select to select from (if set, FromTable must not be set)
	SpineSelect          *SelectNode      // Sub-select that returns a spine of dimensions. Currently it will be right-joined onto FromSelect.
	LeftJoinSelects      []*SelectNode    // Sub-selects to left join onto FromSelect, to enable "per-dimension" measures
	JoinComparisonSelect *SelectNode      // Sub-select to join onto FromSelect for comparison measures
	JoinComparisonType   JoinType         // Type of join to use for JoinComparisonSelect
	Unnests              []string         // Unnest expressions to add in the FROM clause
	Group                bool             // Whether the SELECT is grouped. If yes, it will group by all DimFields.
	Where                *ExprNode        // Expression for the WHERE clause
	TimeWhere            *ExprNode        // Expression for the time range to add to the WHERE clause
	Having               *ExprNode        // Expression for the HAVING clause. If HAVING is not allowed in the current context, it will added as a WHERE in a wrapping SELECT.
	OrderBy              []OrderFieldNode // Fields to order by
	Limit                *int64           // Limit for the query
	Offset               *int64           // Offset for the query
}

SelectNode represents a query that computes measures by dimensions. The from/join clauses are not all compatible. The allowed combinations are:

• FromTable
• FromSelect and optionally SpineSelect and/or LeftJoinSelects
• FromSelect and optionally JoinComparisonSelect (for comparison CTE based optimization, this combination is used, both should be set and one of them will be used as CTE)

type Sort

type Sort struct {
	Name string `mapstructure:"name"`
	Desc bool   `mapstructure:"desc"`
}

type Spine

type Spine struct {
	Where     *WhereSpine `mapstructure:"where"`
	TimeRange *TimeSpine  `mapstructure:"time"`
}

type Subquery

type Subquery struct {
	Dimension Dimension   `mapstructure:"dimension"`
	Measures  []Measure   `mapstructure:"measures"`
	Where     *Expression `mapstructure:"where"`
	Having    *Expression `mapstructure:"having"`
}

type TimeGrain

type TimeGrain string

const (
	TimeGrainUnspecified TimeGrain = ""
	TimeGrainMillisecond TimeGrain = "millisecond"
	TimeGrainSecond      TimeGrain = "second"
	TimeGrainMinute      TimeGrain = "minute"
	TimeGrainHour        TimeGrain = "hour"
	TimeGrainDay         TimeGrain = "day"
	TimeGrainWeek        TimeGrain = "week"
	TimeGrainMonth       TimeGrain = "month"
	TimeGrainQuarter     TimeGrain = "quarter"
	TimeGrainYear        TimeGrain = "year"
)

func TimeGrainFromProto

func TimeGrainFromProto(t runtimev1.TimeGrain) TimeGrain

func (TimeGrain) ToProto

func (t TimeGrain) ToProto() runtimev1.TimeGrain

func (TimeGrain) ToTimeutil

func (t TimeGrain) ToTimeutil() timeutil.TimeGrain

func (TimeGrain) Valid

func (t TimeGrain) Valid() bool

type TimeRange

type TimeRange struct {
	Start        time.Time `mapstructure:"start"`
	End          time.Time `mapstructure:"end"`
	IsoDuration  string    `mapstructure:"iso_duration"`
	IsoOffset    string    `mapstructure:"iso_offset"`
	RoundToGrain TimeGrain `mapstructure:"round_to_grain"`
}

func (*TimeRange) IsZero

func (tr *TimeRange) IsZero() bool

type TimeSpine

type TimeSpine struct {
	Start time.Time `mapstructure:"start"`
	End   time.Time `mapstructure:"end"`
	Grain TimeGrain `mapstructure:"grain"`
}

type WhereSpine

type WhereSpine struct {
	Expression *Expression `mapstructure:"expr"`
}