pipeline

Documentation

Overview

package pipeline implements a system for running data pipelines on top of the filesystem

Index

• Variables
• func RunPipelines(pipelineDir, inRepo, outRepo, commit, branch, shard string) error
• func WaitPipeline(pipelineDir, pipeline, commit string) error
• type Runner
  • func NewRunner(pipelineDir, inRepo, outPrefix, commit, branch, shard string) *Runner
  • func (r *Runner) Cancel() error
  • func (r *Runner) Inputs() ([]string, error)
  • func (r *Runner) Run() error

Variables

var (
	ErrFailed          = errors.New("pfs: pipeline failed")
	ErrCancelled       = errors.New("pfs: cancelled")
	ErrArgCount        = errors.New("pfs: illegal argument count")
	ErrUnkownKeyword   = errors.New("pfs: unknown keyword")
	ErrUnknownProtocol = errors.New("pfs: unknown protocol")
)

Functions

func RunPipelines

func RunPipelines(pipelineDir, inRepo, outRepo, commit, branch, shard string) error

RunPipelines lets you easily run the Pipelines in one line if you don't care about cancelling them.

func WaitPipeline

func WaitPipeline(pipelineDir, pipeline, commit string) error

WaitPipeline waits for a pipeline to complete. If the pipeline fails ErrFailed is returned.

Types

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

func NewRunner

func NewRunner(pipelineDir, inRepo, outPrefix, commit, branch, shard string) *Runner

func (*Runner) Cancel

func (r *Runner) Cancel() error

func (*Runner) Inputs

func (r *Runner) Inputs() ([]string, error)

Inputs returns all of the inputs for the pipelines.

func (*Runner) Run

func (r *Runner) Run() error

Run runs all of the pipelines it finds in pipelineDir. Returns the first error it encounters.