models

Documentation

Index

• type Code
  • func (c Code) Configure(name string) error
• type Data
  • func NewData() Data
  • func (d Data) Unmarshal(value interface{}) error
• type DataProcessor
• type ETL
• type Job
  • func (j *Job) Configure() error
  • func (j *Job) Execute() error
• type Pipeline
  • func NewPipeline(stages ...DataProcessor) *Pipeline
  • func (p *Pipeline) Run() chan error

Types

type Code

type Code string

Code is the script code

func (Code) Configure

func (c Code) Configure(name string) error

Configure saves code to a file and builds plugin

type Data

type Data []byte

Data is what is passes by pipeline stages

func NewData

func NewData() Data

NewData returns default data

func (Data) Unmarshal

func (d Data) Unmarshal(value interface{}) error

Unmarshal calls json.Unmarshal on Data into value

type DataProcessor

type DataProcessor interface {
	ProcessData(
		input Data,
		outputChan chan Data,
		killChan chan error,
	)
}

DataProcessor is a pipeline stage

type ETL

type ETL interface {
	Extract() DataProcessor
	Transform() DataProcessor
	Load() DataProcessor
}

ETL defines an ETL

type Job

type Job struct {
	Name string
	Code Code
	// contains filtered or unexported fields
}

Job has the ETL description

func (*Job) Configure

func (j *Job) Configure() error

Configure loads the job plugin

func (*Job) Execute

func (j *Job) Execute() error

Execute ...

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline contains stages and executes them in order

func NewPipeline

func NewPipeline(stages ...DataProcessor) *Pipeline

NewPipeline create pipeline with stages

func (*Pipeline) Run

func (p *Pipeline) Run() chan error

Run executes stages and return chan of error at first error on at the end of stages