record

Documentation

Overview

Package scan accepts a bufio.SplitFunc and generalizes batches to non-line oriented input, e.g. XML.

Index

• Variables
• type Processor
  • func NewProcessor(r io.Reader, w io.Writer, f func([]byte) ([]byte, error)) *Processor
  • func (p *Processor) Run() error
  • func (p *Processor) Split(f bufio.SplitFunc)
• type TagSplitter
  • func NewTagSplitter(tag string) *TagSplitter
  • func (ts *TagSplitter) Split(data []byte, atEOF bool) (advance int, token []byte, err error)

Variables

var ErrNestedTagsWithSameNameNotImplemented = errors.New("nested tags not implemented")

Types

type Processor

type Processor struct {
	BatchSize  int
	SplitFunc  bufio.SplitFunc
	NumWorkers int
	Verbose    bool
	R          io.Reader
	W          io.Writer
	F          func([]byte) ([]byte, error)
}

Processor can process lines in parallel.

func NewProcessor

func NewProcessor(r io.Reader, w io.Writer, f func([]byte) ([]byte, error)) *Processor

NewProcessor creates a new line processor.

func (*Processor) Run

func (p *Processor) Run() error

Run starts the workers, crunching through the input.

func (*Processor) Split

func (p *Processor) Split(f bufio.SplitFunc)

Split set the SplitFunc to be used to identify records.

type TagSplitter

type TagSplitter struct {
	BatchSize int // number of elements to collect in one batch
	// contains filtered or unexported fields
}

TagSplitter splits a stream on a given XML element.

func NewTagSplitter

func NewTagSplitter(tag string) *TagSplitter

NewTagSplitter returns a TagSplitter for a given XML element name, given as string.

func (*TagSplitter) Split

func (ts *TagSplitter) Split(data []byte, atEOF bool) (advance int, token []byte, err error)

Split finds elements in the stream and will accumulate them up to a given batch size.