parallel

Documentation

Overview

Package parallel implements helpers for fast processing of line oriented inputs.

Index

• Constants
• type BytesBatch
  • func NewBytesBatch() *BytesBatch
  • func NewBytesBatchCapacity(cap int) *BytesBatch
  • func (bb *BytesBatch) Add(b Record)
  • func (bb *BytesBatch) Reset()
  • func (bb *BytesBatch) Size() int
  • func (bb *BytesBatch) Slice() []Record
• type Processor
  • func NewProcessor(r io.Reader, w io.Writer, f TransformerFunc) *Processor
  • func (p *Processor) Run() error
  • func (p *Processor) RunWorkers(numWorkers int) error
• type Record
• type TransformerFunc

Constants

const Version = "0.1.0"

Version of library.

Types

type BytesBatch

type BytesBatch struct {
	// contains filtered or unexported fields
}

BytesBatch is a slice of byte slices.

func NewBytesBatch

func NewBytesBatch() *BytesBatch

NewBytesBatch creates a new BytesBatch with a given capacity.

func NewBytesBatchCapacity

func NewBytesBatchCapacity(cap int) *BytesBatch

NewBytesBatchCapacity creates a new BytesBatch with a given capacity.

func (*BytesBatch) Add

func (bb *BytesBatch) Add(b Record)

Add adds an element to the batch.

func (*BytesBatch) Reset

func (bb *BytesBatch) Reset()

Reset empties this batch.

func (*BytesBatch) Size

func (bb *BytesBatch) Size() int

Size returns the number of elements in the batch.

func (*BytesBatch) Slice

func (bb *BytesBatch) Slice() []Record

Slice returns a slice of byte slices.

type Processor

type Processor struct {
	BatchSize       int
	RecordSeparator byte
	NumWorkers      int
	SkipEmptyLines  bool
	// contains filtered or unexported fields
}

Processor can process lines in parallel.

func NewProcessor

func NewProcessor(r io.Reader, w io.Writer, f TransformerFunc) *Processor

NewProcessor creates a new line processor.

func (*Processor) Run

func (p *Processor) Run() error

Run starts the workers, crunching through the input.

func (*Processor) RunWorkers

func (p *Processor) RunWorkers(numWorkers int) error

RunWorkers allows to quickly set the number of workers.

type Record

type Record struct {
	// contains filtered or unexported fields
}

Record groups a value and a corresponding line number.

type TransformerFunc

type TransformerFunc func(lineno int64, b []byte) ([]byte, error)

TransformerFunc takes a line number and a slice of bytes and returns a slice of bytes and a an error. A common denominator of functions that transform data.