pipeline

Documentation

Overview

Package pipeline contains a streaming pipeline implementation based on the Gopher Academy article by S. Lampa - Patterns for composable concurrent pipelines in Go (https://blog.gopheracademy.com/advent-2015/composable-pipelines-improvements/)

Index

• Constants
• type DataStreamer
  • func NewDataStreamer(info *Info) *DataStreamer
  • func (proc *DataStreamer) Connect(input []string)
  • func (proc *DataStreamer) Run()
• type FastqHandler
  • func NewFastqHandler(info *Info) *FastqHandler
  • func (proc *FastqHandler) Connect(previous *DataStreamer)
  • func (proc *FastqHandler) Run()
• type Info
• type Minion
  • func (minion *Minion) Finish()
  • func (minion *Minion) Start()
• type Pipeline
  • func NewPipeline() *Pipeline
  • func (Pipeline *Pipeline) AddProcess(proc process)
  • func (Pipeline *Pipeline) AddProcesses(procs ...process)
  • func (Pipeline *Pipeline) GetNumProcesses() int
  • func (Pipeline *Pipeline) Run()
• type SeqMinimizer
  • func NewSeqMinimizer(info *Info) *SeqMinimizer
  • func (proc *SeqMinimizer) Connect(previous *FastqHandler)
  • func (proc *SeqMinimizer) Run()
• type SketchCmd
• type Sketcher
  • func NewSketcher(info *Info) *Sketcher
  • func (proc *Sketcher) Connect(previous *SeqMinimizer)
  • func (proc *Sketcher) Run()

Constants

const BUFFERSIZE int = 64

BUFFERSIZE is the size of the buffer used by the pipeline channels

Types

type DataStreamer

type DataStreamer struct {
	// contains filtered or unexported fields
}

DataStreamer is a pipeline process that streams data from STDIN/file

func NewDataStreamer

func NewDataStreamer(info *Info) *DataStreamer

NewDataStreamer is the constructor

func (*DataStreamer) Connect

func (proc *DataStreamer) Connect(input []string)

Connect is the method to connect the DataStreamer to some data source

func (*DataStreamer) Run

func (proc *DataStreamer) Run()

Run is the method to run this process, which satisfies the pipeline interface

type FastqHandler

type FastqHandler struct {
	// contains filtered or unexported fields
}

FastqHandler is a pipeline process to convert a pipeline to the FASTQ type

func NewFastqHandler

func NewFastqHandler(info *Info) *FastqHandler

NewFastqHandler is the constructor

func (*FastqHandler) Connect

func (proc *FastqHandler) Connect(previous *DataStreamer)

Connect is the method to join the input of this process with the output of a DataStreamer

func (*FastqHandler) Run

func (proc *FastqHandler) Run()

Run is the method to run this process, which satisfies the pipeline interface

type Info

type Info struct {
	Version string
	Sketch  *SketchCmd
}

Info stores the runtime information

type Minion

type Minion struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Minion is the base data type

func (*Minion) Finish

func (minion *Minion) Finish()

Finish is a method to close down a minion, after checking it isn't currently working on something

func (*Minion) Start

func (minion *Minion) Start()

Start is a method to start the minion running

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline is the base type, which takes any types that satisfy the process interface

func NewPipeline

func NewPipeline() *Pipeline

NewPipeline is the pipeline constructor

func (*Pipeline) AddProcess

func (Pipeline *Pipeline) AddProcess(proc process)

AddProcess is a method to add a single process to the pipeline

func (*Pipeline) AddProcesses

func (Pipeline *Pipeline) AddProcesses(procs ...process)

AddProcesses is a method to add multiple processes to the pipeline

func (*Pipeline) GetNumProcesses

func (Pipeline *Pipeline) GetNumProcesses() int

GetNumProcesses is a method to return the number of processes registered in a pipeline

func (*Pipeline) Run

func (Pipeline *Pipeline) Run()

Run is a method that starts the pipeline

type SeqMinimizer

type SeqMinimizer struct {
	// contains filtered or unexported fields
}

SeqMinimizer is a process to collect minimizers from sequences

func NewSeqMinimizer

func NewSeqMinimizer(info *Info) *SeqMinimizer

NewSeqMinimizer is the constructor

func (*SeqMinimizer) Connect

func (proc *SeqMinimizer) Connect(previous *FastqHandler)

Connect is the method to join the input of this process with the output of FastqHandler

func (*SeqMinimizer) Run

func (proc *SeqMinimizer) Run()

Run is the method to run this process, which satisfies the pipeline interface

type SketchCmd

type SketchCmd struct {
	FileName     string // this is the name of the input file(s) which has been sketched, or STDIN if -f was not provided
	Fasta        bool
	KmerSize     uint
	WindowSize   uint
	SpectrumSize int32
	SketchSize   uint
	ChunkSize    uint
	DecayRatio   float64
	Stream       bool
	Interval     uint
	OutFile      string
	NumMinions   int
	BannerLabel  string
	KHF          bool
	KMV          bool
}

SketchCmd stores the runtime info for the sketch command

type Sketcher

type Sketcher struct {
	// contains filtered or unexported fields
}

Sketcher is a pipeline process that receives k-mer spectra data from minions and histosketches it

func NewSketcher

func NewSketcher(info *Info) *Sketcher

NewSketcher is the constructor

func (*Sketcher) Connect

func (proc *Sketcher) Connect(previous *SeqMinimizer)

Connect is the method to join the input of this process with the output of SeqMinimizer

func (*Sketcher) Run

func (proc *Sketcher) Run()

Run is the method to run this process, which satisfies the pipeline interface