pipeline

Documentation

Overview

package pipeline implements a system for running data pipelines on top of the filesystem

Index

• Variables
• func RunPipelines(pipelineDir string, inRepo string, outRepo string, commit string, ...) error
• func WaitPipeline(pipelineDir, pipeline, commit string) error
• type Runner
  • func NewRunner(pipelineDir string, inRepo string, outPrefix string, commit string, ...) *Runner
  • func (r *Runner) Cancel() error
  • func (r *Runner) Inputs() ([]string, error)
  • func (r *Runner) Run() error

Variables

var (
	ErrFailed          = errors.New("pfs: pipeline failed")
	ErrCancelled       = errors.New("pfs: cancelled")
	ErrArgCount        = errors.New("pfs: illegal argument count")
	ErrUnkownKeyword   = errors.New("pfs: unknown keyword")
	ErrUnknownProtocol = errors.New("pfs: unknown protocol")
)

Functions

func RunPipelines

func RunPipelines(pipelineDir string, inRepo string, outRepo string, commit string, branch string, shard string, cache etcache.Cache) error

RunPipelines lets you easily run the Pipelines in one line if you don't care about cancelling them.

func WaitPipeline

func WaitPipeline(pipelineDir, pipeline, commit string) error

WaitPipeline waits for a pipeline to complete. If the pipeline fails ErrFailed is returned.

Types

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

func NewRunner

func NewRunner(pipelineDir string, inRepo string, outPrefix string, commit string, branch string, shard string, cache etcache.Cache) *Runner

func (*Runner) Cancel

func (r *Runner) Cancel() error

func (*Runner) Inputs

func (r *Runner) Inputs() ([]string, error)

Inputs returns all of the inputs for the pipelines.

func (*Runner) Run

func (r *Runner) Run() error

Run runs all of the pipelines it finds in pipelineDir. Returns the first error it encounters.