topdown

package
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 10, 2017 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package topdown provides query evaluation support.

The topdown implementation is a (slightly modified) version of the standard "top-down evaluation" algorithm used in query languages such as Datalog. The main difference in the implementation is in the handling of Rego references.

At a high level, the topdown implementation consists of (1) term evaluation and (2) expression evaluation. Term evaluation involves handling expression terms in isolation whereas expression evaluation involves evaluating the terms in relation to each other.

During the term evaluation phase, the topdown implementation will evaluate references found in the expression to produce bindings for:

  1. Variables appearing in references
  2. References to Virtual Documents
  3. Comprehensions

Once terms have been evaluated in isolation, the overall expression can be evaluated. If the expression is simply a term (e.g., string, reference, variable, etc.) then it is compared against the boolean "false" value. If the value IS NOT "false", evaluation continues. On the other hand, if the expression is defined by a built-in operator (e.g., =, !=, min, max, etc.) then the appropriate built-in function is invoked to determine if evaluation continues and bind variables accordingly.

Index

Examples

Constants

View Source 
const (

	// InternalErr represents an unknown evaluation error.
	InternalErr string = "eval_internal_error"

	// CancelErr indicates the evaluation process was cancelled.
	CancelErr string = "eval_cancel_error"

	// ConflictErr indicates a conflict was encountered during evaluation. For
	// instance, a conflict occurs if a rule produces multiple, differing values
	// for the same key in an object. Conflict errors indicate the policy does
	// not account for the data loaded into the policy engine.
	ConflictErr string = "eval_conflict_error"

	// TypeErr indicates evaluation stopped because an expression was applied to
	// a value of an inappropriate type.
	TypeErr string = "eval_type_error"
)

Variables

This section is empty.

Functions

func Continue 

func Continue(t *Topdown, key, value ast.Value, iter Iterator) error

Continue binds key to value in t and calls the iterator. This is a helper function for simple cases where a single value (e.g., a variable) needs to be bound to a value in order for the evaluation the proceed.

func Eval 

func Eval(t *Topdown, iter Iterator) error

Eval evaluates the query in t and calls iter once for each set of bindings that satisfy all of the expressions in the query.

Example 
package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"

	"github.com/open-policy-agent/opa/ast"
	"github.com/open-policy-agent/opa/storage/inmem"
	"github.com/open-policy-agent/opa/topdown"
)

func main() {
	// Initialize context for the example. Normally the caller would obtain the
	// context from an input parameter or instantiate their own.
	ctx := context.Background()

	compiler := ast.NewCompiler()

	// Define a dummy query and some data that the query will execute against.
	query, err := compiler.QueryCompiler().Compile(ast.MustParseBody(`data.a[_] = x; x >= 2`))
	if err != nil {
		// Handle error.
	}

	var data map[string]interface{}

	// OPA uses Go's standard JSON library but assumes that numbers have been
	// decoded as json.Number instead of float64. You MUST decode with UseNumber
	// enabled.
	decoder := json.NewDecoder(bytes.NewBufferString(`{"a": [1,2,3,4]}`))
	decoder.UseNumber()

	if err := decoder.Decode(&data); err != nil {
		// Handle error.
	}

	// Instantiate the policy engine's storage layer.
	store := inmem.NewFromObject(data)

	// Create a new transaction. Transactions allow the policy engine to
	// evaluate the query over a consistent snapshot fo the storage layer.
	txn, err := store.NewTransaction(ctx)
	if err != nil {
		// Handle error.
	}

	defer store.Abort(ctx, txn)

	// Prepare the evaluation parameters. Evaluation executes against the policy
	// engine's storage. In this case, we seed the storage with a single array
	// of number. Other parameters such as the input, tracing configuration,
	// etc. can be set on the Topdown object.
	t := topdown.New(ctx, query, compiler, store, txn)

	result := []interface{}{}

	// Execute the query and provide a callbakc function to accumulate the results.
	err = topdown.Eval(t, func(t *topdown.Topdown) error {

		// Each variable in the query will have an associated "binding".
		x := t.Binding(ast.Var("x"))

		// Alternatively, you can get a mapping of all bound variables.
		x = t.Vars()[ast.Var("x")]

		// The bindings are ast.Value types so we will convert to a native Go value here.
		v, err := ast.ValueToInterface(x, t)
		if err != nil {
			return err
		}

		result = append(result, v)
		return nil
	})

	// Inspect the query result.
	fmt.Println("result:", result)
	fmt.Println("err:", err)

}

Output:

result: [2 3 4]
err: <nil>

func IsError added in v0.4.5 

func IsError(err error) bool

IsError returns true if the err is an Error.

func MakeInput added in v0.4.0 

func MakeInput(pairs [][2]*ast.Term) (ast.Value, error)

MakeInput converts the slice of key/value pairs into a single input value. The keys define an object hierarchy.

func PlugExpr 

func PlugExpr(expr *ast.Expr, binding Binding) *ast.Expr

PlugExpr returns a copy of expr with bound terms substituted for the binding.

func PlugHead added in v0.2.0 

func PlugHead(head *ast.Head, binding Binding) *ast.Head

PlugHead returns a copy of head with bound terms substituted for the binding.

func PlugTerm 

func PlugTerm(term *ast.Term, binding Binding) *ast.Term

PlugTerm returns a copy of term with bound terms substituted for the binding.

func PlugValue 

func PlugValue(v ast.Value, binding func(ast.Value) ast.Value) ast.Value

PlugValue returns a copy of v with bound terms substituted for the binding.

func PrettyTrace added in v0.2.0 

func PrettyTrace(w io.Writer, trace []*Event)

PrettyTrace pretty prints the trace to the writer.

func RegisterBuiltinFunc 

func RegisterBuiltinFunc(name ast.String, fun BuiltinFunc)

RegisterBuiltinFunc adds a new built-in function to the evaluation engine.

func RegisterFunctionalBuiltin1 added in v0.4.0 

func RegisterFunctionalBuiltin1(name ast.String, fun FunctionalBuiltin1)

RegisterFunctionalBuiltin1 adds a new built-in function to the evaluation engine.

Example 
package main

import (
	"context"
	"fmt"

	"strings"

	"github.com/open-policy-agent/opa/ast"
	"github.com/open-policy-agent/opa/topdown"
	"github.com/open-policy-agent/opa/topdown/builtins"
	"github.com/open-policy-agent/opa/types"
)

func main() {

	// Rego includes a number of built-in functions ("built-ins") for performing
	// standard operations like string manipulation, regular expression
	// matching, and computing aggregates.
	//
	// This test shows how to add a new built-in to Rego and OPA.

	// Initialize context for the example. Normally the caller would obtain the
	// context from an input parameter or instantiate their own.
	ctx := context.Background()

	// The ast package contains a registry that enumerates the built-ins
	// included in Rego. When adding a new built-in, you must update the
	// registry to include your built-in. Otherwise, the compiler will complain
	// when it encounters your built-in.
	builtin := &ast.Builtin{
		Name: ast.String("mybuiltins.upper"),
		Args: []types.Type{
			types.S,
			types.S,
		},
		TargetPos: []int{1},
	}

	ast.RegisterBuiltin(builtin)

	// This is the implementation of the built-in that will be called during
	// query evaluation.
	builtinImpl := func(a ast.Value) (ast.Value, error) {

		str, err := builtins.StringOperand(a, 1)

		if err != nil {
			return nil, err
		}

		if str.Equal(ast.String("magic")) {
			// topdown.BuiltinEmpty indicates to the evaluation engine that the
			// expression is false/not defined.
			return nil, topdown.BuiltinEmpty{}
		}

		return ast.String(strings.ToUpper(string(str))), nil
	}

	// See documentation for registering functions that take different numbers
	// of arguments.
	topdown.RegisterFunctionalBuiltin1(builtin.Name, builtinImpl)

	// At this point, the new built-in has been registered and can be used in
	// queries. Our custom built-in converts strings to upper case but is not
	// defined for the input "magic".
	compiler := ast.NewCompiler()
	query, err := compiler.QueryCompiler().Compile(ast.MustParseBody(`mybuiltins.upper("custom", x); not mybuiltins.upper("magic", "MAGIC")`))
	if err != nil {
		// Handle error.
	}

	// Evaluate the query.
	t := topdown.New(ctx, query, compiler, nil, nil)

	topdown.Eval(t, func(t *topdown.Topdown) error {
		fmt.Println("x:", t.Binding(ast.Var("x")))
		return nil
	})

	// If you are adding new built-in functions to upstream OPA, you must also
	// update the [Language
	// Reference](http://www.openpolicyagent.org/documentation/references/language/)
	// and [How Do I Write
	// Policies](http://www.openpolicyagent.org/documentation/how-do-i-write-policies/)
	// documents. In addition, you must add tests for your new built-in. See the
	// existing integration tests in the topdown package.

}

Output:


x: "CUSTOM"

func RegisterFunctionalBuiltin1Out3 added in v0.4.10 

func RegisterFunctionalBuiltin1Out3(name ast.String, fun FunctionalBuiltin1Out3)

RegisterFunctionalBuiltin1Out3 adds a new built-in function to the evaluation engine.

func RegisterFunctionalBuiltin2 added in v0.4.0 

func RegisterFunctionalBuiltin2(name ast.String, fun FunctionalBuiltin2)

RegisterFunctionalBuiltin2 adds a new built-in function to the evaluation engine.

func RegisterFunctionalBuiltin3 added in v0.4.0 

func RegisterFunctionalBuiltin3(name ast.String, fun FunctionalBuiltin3)

RegisterFunctionalBuiltin3 adds a new built-in function to the evaluation engine.

func RegisterFunctionalBuiltinVoid1 added in v0.5.0 

func RegisterFunctionalBuiltinVoid1(name ast.String, fun FunctionalBuiltinVoid1)

RegisterFunctionalBuiltinVoid1 adds a new built-in function to the evaluation engine.

func RegisterFunctionalBuiltinVoid2 added in v0.4.0 

func RegisterFunctionalBuiltinVoid2(name ast.String, fun FunctionalBuiltinVoid2)

RegisterFunctionalBuiltinVoid2 adds a new built-in function to the evaluation engine.

func ResetQueryIDs added in v0.2.0 

func ResetQueryIDs()

ResetQueryIDs resets the query ID generator. This is only for test purposes.

func ResolveRefs added in v0.2.0 

func ResolveRefs(v ast.Value, t *Topdown) (ast.Value, error)

ResolveRefs returns the value obtained by resolving references to base documents.

func ResolveRefsTerm added in v0.4.5 

func ResolveRefsTerm(term *ast.Term, t *Topdown) (*ast.Term, error)

ResolveRefsTerm returns a copy of term obtained by resolving references to base documents.

Types

type Binding added in v0.2.0 

type Binding func(ast.Value) ast.Value

Binding defines the interface used to apply term bindings to terms, expressions, etc.

type BufferTracer added in v0.2.0 

type BufferTracer []*Event

BufferTracer implements the Tracer interface by simply buffering all events received.

func NewBufferTracer added in v0.2.0 

func NewBufferTracer() *BufferTracer

NewBufferTracer returns a new BufferTracer.

func (*BufferTracer) Enabled added in v0.2.0 

func (b *BufferTracer) Enabled() bool

Enabled always returns true.

func (*BufferTracer) Trace added in v0.2.0 

func (b *BufferTracer) Trace(t *Topdown, evt *Event)

Trace adds the event to the buffer.

type BuiltinEmpty added in v0.4.0 

type BuiltinEmpty struct{}

BuiltinEmpty is used to signal that the built-in function evaluated, but the result is undefined so evaluation should not continue.

func (BuiltinEmpty) Error added in v0.4.0 

func (BuiltinEmpty) Error() string

type BuiltinFunc 

type BuiltinFunc func(t *Topdown, expr *ast.Expr, iter Iterator) (err error)

BuiltinFunc defines the interface that the evaluation engine uses to invoke built-in functions (built-ins). In most cases, custom built-ins can be implemented using the FunctionalBuiltin interfaces (which provide less control but are much simpler).

Users can implement their own built-ins and register them with OPA.

Built-ins are given the current evaluation context t with the expression expr to be evaluated. Built-ins can assume that the expression has been plugged with bindings from the current context however references to base documents will not have been resolved. If the built-in determines that the expression has evaluated successfully it should bind any output variables and invoke the iterator with the context produced by binding the output variables. Built-ins must be sure to unbind the outputs after the iterator returns.

type Cancel added in v0.5.1 

type Cancel interface {
	Cancel()
	Cancelled() bool
}

Cancel defines the interface for cancelling topdown queries. Cancel operations are thread-safe and idempotent.

func NewCancel added in v0.5.1 

func NewCancel() Cancel

NewCancel returns a new Cancel object.

type Error 

type Error struct {
	Code     string        `json:"code"`
	Message  string        `json:"message"`
	Location *ast.Location `json:"location,omitempty"`
}

Error is the error type returned by the Eval and Query functions when an evaluation error occurs.

func (*Error) Error 

func (e *Error) Error() string

type Event added in v0.2.0 

type Event struct {
	Op       Op            // Identifies type of event.
	Node     interface{}   // Contains AST node relevant to the event.
	QueryID  uint64        // Identifies the query this event belongs to.
	ParentID uint64        // Identifies the parent query this event belongs to.
	Locals   *ast.ValueMap // Contains local variable bindings from the query context.
}

Event contains state associated with a tracing event.

func (*Event) Equal added in v0.2.0 

func (evt *Event) Equal(other *Event) bool

Equal returns true if this event is equal to the other event.

func (*Event) HasBody added in v0.2.0 

func (evt *Event) HasBody() bool

HasBody returns true if the Event contains an ast.Body.

func (*Event) HasExpr added in v0.2.0 

func (evt *Event) HasExpr() bool

HasExpr returns true if the Event contains an ast.Expr.

func (*Event) HasRule added in v0.2.0 

func (evt *Event) HasRule() bool

HasRule returns true if the Event contains an ast.Rule.

func (*Event) String added in v0.2.0 

func (evt *Event) String() string

type FunctionalBuiltin1 added in v0.4.0 

type FunctionalBuiltin1 func(op1 ast.Value) (output ast.Value, err error)

FunctionalBuiltin1 defines an interface for simple functional built-ins.

Implement this interface if your built-in function takes one input and produces one output.

If an error occurs, the functional built-in should return a descriptive message. The message should not be prefixed with the built-in name as the framework takes care of this.

type FunctionalBuiltin1Out3 added in v0.4.10 

type FunctionalBuiltin1Out3 func(op1 ast.Value) (a, b, c ast.Value, err error)

FunctionalBuiltin1Out3 defines an interface for functional built-ins.

Implement this interface if your built-in function takes one input and produces three outputs.

If an error occurs, the functional built-in should return a descriptive message. The message should not be prefixed with the built-in name as the framework takes care of this.

type FunctionalBuiltin2 added in v0.4.0 

type FunctionalBuiltin2 func(op1, op2 ast.Value) (output ast.Value, err error)

FunctionalBuiltin2 defines an interface for simple functional built-ins.

Implement this interface if your built-in function takes two inputs and produces one output.

If an error occurs, the functional built-in should return a descriptive message. The message should not be prefixed with the built-in name as the framework takes care of this.

type FunctionalBuiltin3 added in v0.4.0 

type FunctionalBuiltin3 func(op1, op2, op3 ast.Value) (output ast.Value, err error)

FunctionalBuiltin3 defines an interface for simple functional built-ins.

Implement this interface if your built-in function takes three inputs and produces one output.

If an error occurs, the functional built-in should return a descriptive message. The message should not be prefixed with the built-in name as the framework takes care of this.

type FunctionalBuiltinVoid1 added in v0.5.0 

type FunctionalBuiltinVoid1 func(op1 ast.Value) (err error)

FunctionalBuiltinVoid1 defines an interface for simple functional built-ins.

Implement this interface if your built-in function takes one input and produces no outputs.

If an error occurs, the functional built-in should return a descriptive message. The message should not be prefixed with the built-in name as the framework takes care of this.

type FunctionalBuiltinVoid2 added in v0.4.0 

type FunctionalBuiltinVoid2 func(op1, op2 ast.Value) (err error)

FunctionalBuiltinVoid2 defines an interface for simple functional built-ins.

Implement this interface if your built-in function takes two inputs and produces no outputs.

If an error occurs, the functional built-in should return a descriptive message. The message should not be prefixed with the built-in name as the framework takes care of this.

type Iterator 

type Iterator func(*Topdown) error

Iterator is the interface for processing evaluation results.

type Op added in v0.2.0 

type Op string

Op defines the types of tracing events. 

const (
	// EnterOp is emitted when a new query is about to be evaluated.
	EnterOp Op = "Enter"

	// ExitOp is emitted when a query has evaluated to true.
	ExitOp Op = "Exit"

	// EvalOp is emitted when an expression is about to be evaluated.
	EvalOp Op = "Eval"

	// RedoOp is emitted when an expression, rule, or query is being re-evaluated.
	RedoOp Op = "Redo"

	// FailOp is emitted when an expression evaluates to false.
	FailOp Op = "Fail"
)

type QueryParams 

type QueryParams struct {
	Context     context.Context
	Cancel      Cancel
	Compiler    *ast.Compiler
	Store       storage.Store
	Transaction storage.Transaction
	Input       ast.Value
	Tracer      Tracer
	Metrics     metrics.Metrics
	Path        ast.Ref
}

QueryParams defines input parameters for the query interface.

func NewQueryParams 

func NewQueryParams(ctx context.Context, compiler *ast.Compiler, store storage.Store, txn storage.Transaction, input ast.Value, path ast.Ref) *QueryParams

NewQueryParams returns a new QueryParams.

func (*QueryParams) NewTopdown added in v0.3.0 

func (q *QueryParams) NewTopdown(query ast.Body) *Topdown

NewTopdown returns a new Topdown object.

This function will not propagate optional values such as the tracer, input document, etc. Those must be set by the caller.

type QueryResult added in v0.2.1 

type QueryResult struct {
	Result   interface{}            // Result contains the document referred to by the params Path.
	Bindings map[string]interface{} // Bindings contains values for variables in the params Input.
}

QueryResult represents a single query result.

func (*QueryResult) String added in v0.2.1 

func (qr *QueryResult) String() string

type QueryResultSet added in v0.2.1 

type QueryResultSet []*QueryResult

QueryResultSet represents a collection of query results.

func Query 

func Query(params *QueryParams) (QueryResultSet, error)

Query returns the value of the document referred to by the params' path. If the params' input contains non-ground terms, there may be multiple query results.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/open-policy-agent/opa/ast"
	"github.com/open-policy-agent/opa/topdown"
)

func main() {
	// Initialize context for the example. Normally the caller would obtain the
	// context from an input parameter or instantiate their own.
	ctx := context.Background()

	compiler := ast.NewCompiler()

	// Define a dummy module with rules that produce documents that we will query below.
	module, err := ast.ParseModule("my_module.rego", `package opa.example

p[x] { q[x]; not r[x] }
q[y] { a = [1, 2, 3]; y = a[_] }
r[z] { b = [2, 4]; z = b[_] }`,
	)

	mods := map[string]*ast.Module{
		"my_module": module,
	}

	if compiler.Compile(mods); compiler.Failed() {
		fmt.Println(compiler.Errors)
	}

	if err != nil {
		// Handle error.
	}

	// Prepare query parameters. In this case, there are no additional documents
	// required by the policy so the input is nil.
	var input ast.Value
	params := topdown.NewQueryParams(ctx, compiler, nil, nil, input, ast.MustParseRef("data.opa.example.p"))

	// Execute the query against "p".
	v1, err1 := topdown.Query(params)

	// Inspect the result.
	fmt.Println("v1:", v1[0].Result)
	fmt.Println("err1:", err1)

}

Output:

v1: [1 3]
err1: <nil>

func (*QueryResultSet) Add added in v0.2.1 

func (qrs *QueryResultSet) Add(qr *QueryResult)

Add inserts a result into the query result set.

func (QueryResultSet) Undefined added in v0.2.1 

func (qrs QueryResultSet) Undefined() bool

Undefined returns true if the query did not find any results.

type Resolver added in v0.2.1 

type Resolver interface {
	Resolve(ref ast.Ref) (value interface{}, err error)
}

Resolver defines the interface for resolving references to base documents to native Go values. The native Go value types map to JSON types.

type Topdown added in v0.3.0 

type Topdown struct {
	Cancel   Cancel
	Query    ast.Body
	Compiler *ast.Compiler
	Input    ast.Value
	Index    int
	Previous *Topdown
	Store    storage.Store
	Tracer   Tracer
	Context  context.Context
	// contains filtered or unexported fields
}

Topdown stores the state of the evaluation process and contains context needed to evaluate queries.

func New added in v0.3.0 

func New(ctx context.Context, query ast.Body, compiler *ast.Compiler, store storage.Store, txn storage.Transaction) *Topdown

New returns a new Topdown object without any bindings.

func (*Topdown) Bind added in v0.3.0 

func (t *Topdown) Bind(key ast.Value, value ast.Value, prev *Undo) *Undo

Bind updates t to include a binding from the key to the value. The return value is used to return t to the state before the binding was added.

func (*Topdown) Binding added in v0.3.0 

func (t *Topdown) Binding(k ast.Value) ast.Value

Binding returns the value bound to the given key.

func (*Topdown) Child added in v0.3.0 

func (t *Topdown) Child(query ast.Body) *Topdown

Child returns a new Topdown object to evaluate query without bindings from t.

func (*Topdown) Closure added in v0.4.1 

func (t *Topdown) Closure(query ast.Body) *Topdown

Closure returns a new Topdown object to evaluate query with bindings from t.

func (*Topdown) Current added in v0.3.0 

func (t *Topdown) Current() *ast.Expr

Current returns the current expression to evaluate.

func (*Topdown) Resolve added in v0.3.0 

func (t *Topdown) Resolve(ref ast.Ref) (interface{}, error)

Resolve returns the native Go value referred to by the ref.

func (*Topdown) Step added in v0.3.0 

func (t *Topdown) Step() *Topdown

Step returns a new Topdown object to evaluate the next expression.

func (*Topdown) Unbind added in v0.3.0 

func (t *Topdown) Unbind(undo *Undo)

Unbind updates t by removing the binding represented by the undo.

func (*Topdown) Vars added in v0.4.0 

func (t *Topdown) Vars() map[ast.Var]ast.Value

Vars returns bindings for the vars in the current query

func (*Topdown) WithCancel added in v0.5.1 

func (t *Topdown) WithCancel(c Cancel) *Topdown

WithCancel returns a new Topdown object that has cancellation set.

func (*Topdown) WithInput added in v0.4.1 

func (t *Topdown) WithInput(input ast.Value) *Topdown

WithInput returns a new Topdown object that has the input document set.

func (*Topdown) WithTracer added in v0.4.1 

func (t *Topdown) WithTracer(tracer Tracer) *Topdown

WithTracer returns a new Topdown object that has a tracer set.

type Tracer 

type Tracer interface {
	Enabled() bool
	Trace(t *Topdown, evt *Event)
}

Tracer defines the interface for tracing in the top-down evaluation engine.

type Undo 

type Undo struct {
	Key   ast.Value
	Value ast.Value
	Prev  *Undo
}

Undo represents a binding that can be undone.

type Vars added in v0.4.0 

type Vars map[ast.Var]ast.Value

Vars represents a set of var bindings.

func (Vars) Diff added in v0.4.0 

func (vs Vars) Diff(other Vars) Vars

Diff returns the var bindings in vs that are not in other.

func (Vars) Equal added in v0.4.0 

func (vs Vars) Equal(other Vars) bool

Equal returns true if vs is equal to other.

Source Files

View all Source files

Directories

Path Synopsis
Package builtins contains utilities for implementing built-in functions.
 Package builtins contains utilities for implementing built-in functions.
Package explain contains utilities for post-processing traces emitted by the evaluation engine.
 Package explain contains utilities for post-processing traces emitted by the evaluation engine.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.