Documentation ¶
Index ¶
- Constants
- Variables
- func ComputeBounds(node Node) error
- func Formatted(p *Spec, opts ...FormatOption) fmt.Formatter
- func HasSideEffect(spec ProcedureSpec) bool
- func RegisterLogicalRules(rules ...Rule)
- func RegisterPhysicalRules(rules ...Rule)
- func RegisterProcedureSpec(k ProcedureKind, c CreateProcedureSpec, qks ...flux.OperationKind)
- func RegisterProcedureSpecWithSideEffect(k ProcedureKind, c CreateProcedureSpec, qks ...flux.OperationKind)
- func ReplaceNode(oldNode, newNode Node)
- func SetTriggerSpec(node Node) error
- func WalkPredecessors(roots []Node, visitFn func(node Node) error) error
- func WalkSuccessors(roots []Node, visitFn func(node Node) error) error
- type Administration
- type AfterAtLeastCountTriggerSpec
- type AfterProcessingTimeTriggerSpec
- type AfterWatermarkTriggerSpec
- type AnyPattern
- type Bounds
- type BoundsAwareProcedureSpec
- type Cost
- type CreateProcedureSpec
- type DefaultCost
- type FormatOption
- type GeneratedYieldProcedureSpec
- type LogicalNode
- func (e *LogicalNode) AddPredecessors(nodes ...Node)
- func (e *LogicalNode) AddSuccessors(nodes ...Node)
- func (b *LogicalNode) Bounds() *Bounds
- func (e *LogicalNode) ClearPredecessors()
- func (e *LogicalNode) ClearSuccessors()
- func (lpn *LogicalNode) ID() NodeID
- func (lpn *LogicalNode) Kind() ProcedureKind
- func (e *LogicalNode) Predecessors() []Node
- func (lpn *LogicalNode) ProcedureSpec() ProcedureSpec
- func (lpn *LogicalNode) ReplaceSpec(newSpec ProcedureSpec) error
- func (b *LogicalNode) SetBounds(bounds *Bounds)
- func (lpn *LogicalNode) ShallowCopy() Node
- func (e *LogicalNode) Successors() []Node
- type LogicalOption
- type LogicalPlanner
- type NarrowTransformationTriggerSpec
- type Node
- type NodeID
- type OneKindPattern
- type OrFinallyTriggerSpec
- type Pattern
- type PhysicalAttributes
- type PhysicalOneKindPattern
- type PhysicalOption
- type PhysicalPlanNode
- func (e *PhysicalPlanNode) AddPredecessors(nodes ...Node)
- func (e *PhysicalPlanNode) AddSuccessors(nodes ...Node)
- func (b *PhysicalPlanNode) Bounds() *Bounds
- func (e *PhysicalPlanNode) ClearPredecessors()
- func (e *PhysicalPlanNode) ClearSuccessors()
- func (ppn *PhysicalPlanNode) Cost(inStats []Statistics) (cost Cost, outStats Statistics)
- func (ppn *PhysicalPlanNode) ID() NodeID
- func (ppn *PhysicalPlanNode) Kind() ProcedureKind
- func (e *PhysicalPlanNode) Predecessors() []Node
- func (ppn *PhysicalPlanNode) ProcedureSpec() ProcedureSpec
- func (ppn *PhysicalPlanNode) ReplaceSpec(newSpec ProcedureSpec) error
- func (b *PhysicalPlanNode) SetBounds(bounds *Bounds)
- func (ppn *PhysicalPlanNode) ShallowCopy() Node
- func (e *PhysicalPlanNode) Successors() []Node
- type PhysicalPlanner
- type PhysicalProcedureSpec
- type Planner
- type PlannerBuilder
- type PostPhysicalValidator
- type ProcedureKind
- type ProcedureSpec
- type RepeatedTriggerSpec
- type Rule
- type Spec
- type Statistics
- type TriggerAwareProcedureSpec
- type TriggerKind
- type TriggerSpec
- type WindowSpec
- type YieldProcedureSpec
Constants ¶
const AnyKind = "*** any procedure kind ***"
const DefaultYieldName = "_result"
DefaultYieldName is the name of a result that doesn't have any name assigned.
Variables ¶
var DefaultTriggerSpec = AfterWatermarkTriggerSpec{}
EmptyBounds is a time range containing only a single point
Functions ¶
func ComputeBounds ¶
ComputeBounds computes the time bounds for a plan node from the bounds of its predecessors.
func Formatted ¶
func Formatted(p *Spec, opts ...FormatOption) fmt.Formatter
TODO(cwolff): enhance the this output to make it more useful
func HasSideEffect ¶ added in v0.8.0
func HasSideEffect(spec ProcedureSpec) bool
func RegisterLogicalRules ¶
func RegisterLogicalRules(rules ...Rule)
RegisterLogicalRules registers the rule created by createFn with the logical plan.
func RegisterPhysicalRules ¶
func RegisterPhysicalRules(rules ...Rule)
RegisterPhysicalRules registers the rule created by createFn with the physical plan.
func RegisterProcedureSpec ¶
func RegisterProcedureSpec(k ProcedureKind, c CreateProcedureSpec, qks ...flux.OperationKind)
RegisterProcedureSpec registers a new procedure with the specified kind. The call panics if the kind is not unique.
func RegisterProcedureSpecWithSideEffect ¶
func RegisterProcedureSpecWithSideEffect(k ProcedureKind, c CreateProcedureSpec, qks ...flux.OperationKind)
RegisterProcedureSpecWithSideEffect registers a new procedure that produces side effects
func ReplaceNode ¶ added in v0.10.0
func ReplaceNode(oldNode, newNode Node)
ReplaceNode accepts two nodes and attaches all the predecessors of the old node to the new node.
S1 S2 S1 S2 \ / oldNode => newNode / \ / \ P1 P2 P1 P2
As is convention, newNode will not have any successors attached. The planner will take care of this.
func SetTriggerSpec ¶ added in v0.22.0
func WalkPredecessors ¶ added in v0.10.0
WalkPredecessor visits every node in the plan rooted at `roots` in topological order, following predecessor links. If a cycle is detected, no node is visited and an error is returned.
Types ¶
type Administration ¶
type AfterAtLeastCountTriggerSpec ¶ added in v0.22.0
type AfterAtLeastCountTriggerSpec struct {
Count int
}
func (AfterAtLeastCountTriggerSpec) Kind ¶ added in v0.22.0
func (AfterAtLeastCountTriggerSpec) Kind() TriggerKind
type AfterProcessingTimeTriggerSpec ¶ added in v0.22.0
func (AfterProcessingTimeTriggerSpec) Kind ¶ added in v0.22.0
func (AfterProcessingTimeTriggerSpec) Kind() TriggerKind
type AfterWatermarkTriggerSpec ¶ added in v0.22.0
func (AfterWatermarkTriggerSpec) Kind ¶ added in v0.22.0
func (AfterWatermarkTriggerSpec) Kind() TriggerKind
type AnyPattern ¶
type AnyPattern struct{}
AnyPattern describes (and matches) any plan node
func (AnyPattern) Match ¶
func (AnyPattern) Match(node Node) bool
func (AnyPattern) Root ¶
func (AnyPattern) Root() ProcedureKind
type Bounds ¶
Bounds is a range of time
func (*Bounds) Intersect ¶
Intersect returns the intersection of two bounds. It returns empty bounds if one of the input bounds are empty.
type BoundsAwareProcedureSpec ¶
BoundsAwareProcedureSpec is any procedure that modifies the time bounds of its data.
type CreateProcedureSpec ¶
type CreateProcedureSpec func(flux.OperationSpec, Administration) (ProcedureSpec, error)
CreateProcedureSpec creates a ProcedureSpec from an OperationSpec and Administration
type DefaultCost ¶
type DefaultCost struct { }
func (DefaultCost) Cost ¶
func (c DefaultCost) Cost(inStats []Statistics) (Cost, Statistics)
type FormatOption ¶
type FormatOption func(*formatter)
type GeneratedYieldProcedureSpec ¶
type GeneratedYieldProcedureSpec struct { DefaultCost Name string }
GeneratedYieldProcedureSpec provides a special planner-generated yield for queries that don't have explicit calls to yield().
func (*GeneratedYieldProcedureSpec) Copy ¶
func (y *GeneratedYieldProcedureSpec) Copy() ProcedureSpec
func (*GeneratedYieldProcedureSpec) Kind ¶
func (y *GeneratedYieldProcedureSpec) Kind() ProcedureKind
func (*GeneratedYieldProcedureSpec) YieldName ¶
func (y *GeneratedYieldProcedureSpec) YieldName() string
type LogicalNode ¶ added in v0.24.0
type LogicalNode struct { Spec ProcedureSpec // contains filtered or unexported fields }
LogicalNode consists of the input and output edges and a procedure spec that describes what the node does.
func CreateLogicalNode ¶
func CreateLogicalNode(id NodeID, spec ProcedureSpec) *LogicalNode
CreateLogicalNode creates a single logical plan node from a procedure spec. The newly created logical node has no incoming or outgoing edges.
func (*LogicalNode) AddPredecessors ¶ added in v0.24.0
func (e *LogicalNode) AddPredecessors(nodes ...Node)
func (*LogicalNode) AddSuccessors ¶ added in v0.24.0
func (e *LogicalNode) AddSuccessors(nodes ...Node)
func (*LogicalNode) ClearPredecessors ¶ added in v0.24.0
func (e *LogicalNode) ClearPredecessors()
func (*LogicalNode) ClearSuccessors ¶ added in v0.24.0
func (e *LogicalNode) ClearSuccessors()
func (*LogicalNode) ID ¶ added in v0.24.0
func (lpn *LogicalNode) ID() NodeID
ID returns a human-readable identifier unique to this plan.
func (*LogicalNode) Kind ¶ added in v0.24.0
func (lpn *LogicalNode) Kind() ProcedureKind
Kind returns the kind of procedure performed by this plan node.
func (*LogicalNode) Predecessors ¶ added in v0.24.0
func (e *LogicalNode) Predecessors() []Node
func (*LogicalNode) ProcedureSpec ¶ added in v0.24.0
func (lpn *LogicalNode) ProcedureSpec() ProcedureSpec
ProcedureSpec returns the procedure spec for this plan node.
func (*LogicalNode) ReplaceSpec ¶ added in v0.24.0
func (lpn *LogicalNode) ReplaceSpec(newSpec ProcedureSpec) error
func (*LogicalNode) ShallowCopy ¶ added in v0.24.0
func (lpn *LogicalNode) ShallowCopy() Node
func (*LogicalNode) Successors ¶ added in v0.24.0
func (e *LogicalNode) Successors() []Node
type LogicalOption ¶
type LogicalOption interface {
// contains filtered or unexported methods
}
LogicalOption is an option to configure the behavior of the logical plan.
func AddLogicalRules ¶ added in v0.26.0
func AddLogicalRules(rules ...Rule) LogicalOption
func DisableIntegrityChecks ¶ added in v0.10.0
func DisableIntegrityChecks() LogicalOption
Disables integrity checks in the logical planner
func OnlyLogicalRules ¶
func OnlyLogicalRules(rules ...Rule) LogicalOption
OnlyLogicalRules produces a logical plan option that forces only a set of particular rules to be applied.
func RemoveLogicalRules ¶ added in v0.48.0
func RemoveLogicalRules(rules ...string) LogicalOption
type LogicalPlanner ¶
type LogicalPlanner interface { CreateInitialPlan(spec *flux.Spec) (*Spec, error) Plan(*Spec) (*Spec, error) }
LogicalPlanner translates a flux.Spec into a plan.Spec and applies any registered logical rules to the plan.
Logical planning should transform the plan in ways that are independent of actual physical algorithms used to implement operations, and independent of the actual data being processed.
func NewLogicalPlanner ¶
func NewLogicalPlanner(options ...LogicalOption) LogicalPlanner
NewLogicalPlanner returns a new logical plan with the given options. The plan will be configured to apply any logical rules that have been registered.
type NarrowTransformationTriggerSpec ¶ added in v0.22.0
type NarrowTransformationTriggerSpec struct{}
func (NarrowTransformationTriggerSpec) Kind ¶ added in v0.22.0
func (NarrowTransformationTriggerSpec) Kind() TriggerKind
type Node ¶ added in v0.24.0
type Node interface { // Returns an identifier for this plan node ID() NodeID // Returns the time bounds for this plan node Bounds() *Bounds // Plan nodes executed immediately before this node Predecessors() []Node // Plan nodes executed immediately after this node Successors() []Node // Specification of the procedure represented by this node ProcedureSpec() ProcedureSpec // Replaces the procedure spec of this node with another ReplaceSpec(ProcedureSpec) error // Type of procedure represented by this node Kind() ProcedureKind // Helper methods for manipulating a plan // These methods are used during planning SetBounds(bounds *Bounds) AddSuccessors(...Node) AddPredecessors(...Node) ClearSuccessors() ClearPredecessors() ShallowCopy() Node }
Node defines the common interface for interacting with logical and physical plan nodes.
func MergeToLogicalNode ¶ added in v0.24.0
func MergeToLogicalNode(top, bottom Node, procSpec ProcedureSpec) (Node, error)
MergeToLogicalNode merges top and bottom plan nodes into a new plan node, with the given procedure spec.
V1 V2 V1 V2 <-- successors \ / top mergedNode | ==> | bottom W | W
The returned node will have its predecessors set to the predecessors of "bottom", however, it's successors will not be set---it will be the responsibility of the plan to attach the merged node to its successors.
func MergeToPhysicalNode ¶ added in v0.24.0
func MergeToPhysicalNode(top, bottom Node, procSpec PhysicalProcedureSpec) (Node, error)
func SwapPlanNodes ¶
SwapPlanNodes swaps two plan nodes and returns an equivalent sub-plan with the nodes swapped.
V1 V2 V1 V2 \ / A B | ==> | B copy of A | | W W
Note that successors of the original top node will not be updated, and the returned plan node will have no successors. It will be the responsibility of the plan to attach the swapped nodes to successors.
type OneKindPattern ¶
type OneKindPattern struct {
// contains filtered or unexported fields
}
OneKindPattern matches a specified procedure with a predecessor pattern
ProcedureKind / | ... \ pattern1 pattern2 ... patternK
func (OneKindPattern) Match ¶
func (okp OneKindPattern) Match(node Node) bool
func (OneKindPattern) Root ¶
func (okp OneKindPattern) Root() ProcedureKind
type OrFinallyTriggerSpec ¶ added in v0.22.0
type OrFinallyTriggerSpec struct { Main TriggerSpec Finally TriggerSpec }
func (OrFinallyTriggerSpec) Kind ¶ added in v0.22.0
func (OrFinallyTriggerSpec) Kind() TriggerKind
type Pattern ¶
type Pattern interface { Root() ProcedureKind Match(Node) bool }
Pattern represents an operator tree pattern It can match itself against a query plan
func Pat ¶
func Pat(kind ProcedureKind, predecessors ...Pattern) Pattern
Pat returns a pattern that can match a plan node with the given ProcedureKind and whose predecessors match the given predecessor patterns.
For example, to construct a pattern that matches a join followed by a sum:
sum | |X| <=> join(A, B) |> sum() <=> Pat(SumKind, Pat(JoinKind, Any(), Any())) / \
A B
func PhysPat ¶ added in v0.22.0
func PhysPat(kind ProcedureKind, predecessors ...Pattern) Pattern
PhysPat returns a pattern that matches a physical plan node with the given ProcedureKind and whose predecessors match the given predecessor patterns.
type PhysicalAttributes ¶
type PhysicalAttributes struct { }
PhysicalAttributes encapsulates sny physical attributes of the result produced by a physical plan node, such as collation, etc.
type PhysicalOneKindPattern ¶ added in v0.22.0
type PhysicalOneKindPattern struct {
// contains filtered or unexported fields
}
PhysicalOneKindPattern matches a physical operator pattern
func (PhysicalOneKindPattern) Match ¶ added in v0.22.0
func (p PhysicalOneKindPattern) Match(node Node) bool
func (PhysicalOneKindPattern) Root ¶ added in v0.22.0
func (p PhysicalOneKindPattern) Root() ProcedureKind
type PhysicalOption ¶
type PhysicalOption interface {
// contains filtered or unexported methods
}
PhysicalOption is an option to configure the behavior of the physical plan.
func DisableValidation ¶ added in v0.10.0
func DisableValidation() PhysicalOption
Disables validation in the physical planner
func OnlyPhysicalRules ¶
func OnlyPhysicalRules(rules ...Rule) PhysicalOption
OnlyPhysicalRules produces a physical plan option that forces only a particular set of rules to be applied.
func RemovePhysicalRules ¶ added in v0.48.0
func RemovePhysicalRules(rules ...string) PhysicalOption
func WithDefaultMemoryLimit ¶
func WithDefaultMemoryLimit(memBytes int64) PhysicalOption
WithDefaultMemoryLimit sets the default memory limit for plans generated by the plan. If the query spec explicitly sets a memory limit, that limit is used instead of the default.
type PhysicalPlanNode ¶
type PhysicalPlanNode struct { Spec PhysicalProcedureSpec // The trigger spec defines how and when a transformation // sends its tables to downstream operators TriggerSpec TriggerSpec // The attributes required from inputs to this node RequiredAttrs []PhysicalAttributes // The attributes provided to consumers of this node's output OutputAttrs PhysicalAttributes // contains filtered or unexported fields }
PhysicalPlanNode represents a physical operation in a plan.
func CreatePhysicalNode ¶
func CreatePhysicalNode(id NodeID, spec PhysicalProcedureSpec) *PhysicalPlanNode
CreatePhysicalNode creates a single physical plan node from a procedure spec. The newly created physical node has no incoming or outgoing edges.
func (*PhysicalPlanNode) AddPredecessors ¶
func (e *PhysicalPlanNode) AddPredecessors(nodes ...Node)
func (*PhysicalPlanNode) AddSuccessors ¶
func (e *PhysicalPlanNode) AddSuccessors(nodes ...Node)
func (*PhysicalPlanNode) ClearPredecessors ¶
func (e *PhysicalPlanNode) ClearPredecessors()
func (*PhysicalPlanNode) ClearSuccessors ¶
func (e *PhysicalPlanNode) ClearSuccessors()
func (*PhysicalPlanNode) Cost ¶
func (ppn *PhysicalPlanNode) Cost(inStats []Statistics) (cost Cost, outStats Statistics)
Cost provides the self-cost (i.e., does not include the cost of its predecessors) for this plan node. Caller must provide statistics of predecessors to this node.
func (*PhysicalPlanNode) ID ¶
func (ppn *PhysicalPlanNode) ID() NodeID
ID returns a human-readable id for this plan node.
func (*PhysicalPlanNode) Kind ¶
func (ppn *PhysicalPlanNode) Kind() ProcedureKind
Kind returns the procedure kind for this plan node.
func (*PhysicalPlanNode) Predecessors ¶
func (e *PhysicalPlanNode) Predecessors() []Node
func (*PhysicalPlanNode) ProcedureSpec ¶
func (ppn *PhysicalPlanNode) ProcedureSpec() ProcedureSpec
ProcedureSpec returns the procedure spec for this plan node.
func (*PhysicalPlanNode) ReplaceSpec ¶
func (ppn *PhysicalPlanNode) ReplaceSpec(newSpec ProcedureSpec) error
func (*PhysicalPlanNode) ShallowCopy ¶
func (ppn *PhysicalPlanNode) ShallowCopy() Node
func (*PhysicalPlanNode) Successors ¶
func (e *PhysicalPlanNode) Successors() []Node
type PhysicalPlanner ¶
PhysicalPlanner performs transforms a logical plan to a physical plan, by applying any registered physical rules.
func NewPhysicalPlanner ¶
func NewPhysicalPlanner(options ...PhysicalOption) PhysicalPlanner
NewPhysicalPlanner creates a new physical plan with the specified options. The new plan will be configured to apply any physical rules that have been registered.
type PhysicalProcedureSpec ¶
type PhysicalProcedureSpec interface { Kind() ProcedureKind Copy() ProcedureSpec Cost(inStats []Statistics) (cost Cost, outStats Statistics) }
PhysicalProcedureSpec is similar to its logical counterpart but must provide a method to determine cost.
type PlannerBuilder ¶ added in v0.24.0
type PlannerBuilder struct {
// contains filtered or unexported fields
}
PlannerBuilder provides clients with an easy way to create planners.
func (*PlannerBuilder) AddLogicalOptions ¶ added in v0.24.0
func (pb *PlannerBuilder) AddLogicalOptions(lopt ...LogicalOption)
AddLogicalOptions lets callers specify attributes of the logical planner that will be part of the created planner.
func (*PlannerBuilder) AddPhysicalOptions ¶ added in v0.26.0
func (pb *PlannerBuilder) AddPhysicalOptions(popt ...PhysicalOption)
AddPhysicalOptions lets callers specify attributes of the physical planner that will be part of the created planner.
func (PlannerBuilder) Build ¶ added in v0.24.0
func (pb PlannerBuilder) Build() Planner
Build builds a planner with specified attributes.
type PostPhysicalValidator ¶
PostPhysicalValidator provides an interface that can be implemented by PhysicalProcedureSpecs for any validation checks to be performed post-physical planning.
type ProcedureSpec ¶
type ProcedureSpec interface { Kind() ProcedureKind Copy() ProcedureSpec }
ProcedureSpec specifies a query operation
type RepeatedTriggerSpec ¶ added in v0.22.0
type RepeatedTriggerSpec struct {
Trigger TriggerSpec
}
func (RepeatedTriggerSpec) Kind ¶ added in v0.22.0
func (RepeatedTriggerSpec) Kind() TriggerKind
type Rule ¶
type Rule interface { // The name of this rule (must be unique) Name() string // Pattern for this rule to match against Pattern() Pattern // Rewrite an operation into an equivalent one // The returned node is the new root of the sub tree. // The boolean return value should be true if anything changed during the rewrite. Rewrite(Node) (Node, bool, error) }
Rule is transformation rule for a query operation
type Spec ¶ added in v0.24.0
type Spec struct { Roots map[Node]struct{} Resources flux.ResourceManagement Now time.Time }
Spec holds the result nodes of a query plan with associated metadata
func (*Spec) BottomUpWalk ¶ added in v0.24.0
BottomUpWalk will execute f for each plan node in the Spec, starting from the sources, and only visiting a node after all its predecessors have been visited.
func (*Spec) CheckIntegrity ¶ added in v0.24.0
CheckIntegrity checks the integrity of the plan, i.e.:
- node A is predecessor of B iff B is successor of A;
- there is no cycle.
This check only detects this problem (N2 is predecessor of R, but not viceversa):
N1 <----> R | N2 <-------
And this one (R is successor of N2, but not viceversa):
N1 <------- | |--> R N2 --------
But not this one, because N2 is not reachable from R (root):
N1 <------- |--> R N2 --------
func (*Spec) TopDownWalk ¶ added in v0.24.0
TopDownWalk will execute f for each plan node in the Spec. It always visits a node before visiting its predecessors.
type Statistics ¶
type TriggerAwareProcedureSpec ¶ added in v0.22.0
type TriggerAwareProcedureSpec interface {
TriggerSpec() TriggerSpec
}
type TriggerKind ¶ added in v0.22.0
type TriggerKind int
const ( NarrowTransformation TriggerKind = iota AfterWatermark Repeated AfterProcessingTime AfterAtLeastCount OrFinally )
type TriggerSpec ¶ added in v0.22.0
type TriggerSpec interface {
Kind() TriggerKind
}
type WindowSpec ¶
type YieldProcedureSpec ¶
type YieldProcedureSpec interface {
YieldName() string
}
YieldProcedureSpec is a special procedure that has the side effect of returning a result to the client.