planner

Documentation

Index

• type InputSpec
• type LinkSpec
• type LogicalPlan
• type OutputSpec
• type PhysicalPlan
  • func (p *PhysicalPlan) AddProcessor(processor ProcessorSpec)
  • func (p *PhysicalPlan) ToSubtaskMetas(ctx PlanCtx, step proto.Step) ([][]byte, error)
• type PipelineSpec
• type PlanCtx
• type Planner
  • func NewPlanner() *Planner
  • func (*Planner) Run(planCtx PlanCtx, plan LogicalPlan) (int64, error)
• type ProcessorSpec

Types

type InputSpec

type InputSpec struct {
	ColumnTypes []byte
	Links       []LinkSpec
}

InputSpec is the specification of an input.

type LinkSpec

type LinkSpec struct {
	ProcessorID int
}

LinkSpec is the specification of a link. Link connects pipelines between different nodes.

type LogicalPlan

type LogicalPlan interface {
	ToTaskMeta() ([]byte, error)
	FromTaskMeta([]byte) error
	ToPhysicalPlan(PlanCtx) (*PhysicalPlan, error)
}

LogicalPlan represents a logical plan in distribute framework. A normal flow of distribute framework is: logical plan -> physical plan -> pipelines. To integrate with current distribute framework, the flow becomes: logical plan -> task meta -> physical plan -> subtaskmetas -> pipelines.

type OutputSpec

type OutputSpec struct {
	Links []LinkSpec
}

OutputSpec is the specification of an output.

type PhysicalPlan

type PhysicalPlan struct {
	Processors []ProcessorSpec
}

PhysicalPlan is a DAG of processors in distribute framework. Each processor is a node process the task with a pipeline, and receive/pass the result to other processors via input and output links.

func (*PhysicalPlan) AddProcessor

func (p *PhysicalPlan) AddProcessor(processor ProcessorSpec)

AddProcessor adds a node to the DAG.

func (*PhysicalPlan) ToSubtaskMetas

func (p *PhysicalPlan) ToSubtaskMetas(ctx PlanCtx, step proto.Step) ([][]byte, error)

ToSubtaskMetas converts the physical plan to a list of subtask metas.

type PipelineSpec

type PipelineSpec interface {
	// ToSubtaskMeta converts the pipeline to a subtask meta
	ToSubtaskMeta(PlanCtx) ([]byte, error)
}

PipelineSpec is the specification of an pipeline.

type PlanCtx

type PlanCtx struct {
	Ctx context.Context

	// integrate with current distribute framework
	SessionCtx sessionctx.Context
	TaskID     int64
	TaskKey    string
	TaskType   proto.TaskType
	ThreadCnt  int

	// PreviousSubtaskMetas is subtask metas of previous steps.
	// We can remove this field if we find a better way to pass the result between steps.
	PreviousSubtaskMetas map[proto.Step][][]byte
	GlobalSort           bool
	NextTaskStep         proto.Step
	ExecuteNodesCnt      int

	Store kv.StorageWithPD
}

PlanCtx is the context for planning.

type Planner

type Planner struct{}

Planner represents a distribute plan planner.

func NewPlanner

func NewPlanner() *Planner

NewPlanner creates a new planer instance.

func (*Planner) Run

func (*Planner) Run(planCtx PlanCtx, plan LogicalPlan) (int64, error)

Run runs the distribute plan.

type ProcessorSpec

type ProcessorSpec struct {
	ID       int
	Input    InputSpec
	Pipeline PipelineSpec
	Output   OutputSpec
	// We can remove this field if we find a better way to pass the result between steps.
	Step proto.Step
}

ProcessorSpec is the specification of a processor. A processor is a node in the DAG. It contains input links from other processors, as well as output links to other processors. It also contains an pipeline which is the actual logic of the processor.