zbuf

Documentation

Index

• Constants
• Variables
• func CopyPuller(w zio.Writer, p Puller) error
• func CopyVars(b Batch) []zed.Value
• func NamedScanner(s Scanner, name string) *namedScanner
• func NewComparator(zctx *zed.Context, layout order.Layout) *expr.Comparator
• func NewComparatorNullsMax(zctx *zed.Context, layout order.Layout) *expr.Comparator
• func NoControl(r zio.Reader) *noControl
• func PullerReader(p Puller) zio.Reader
• func WriteBatch(zw zio.Writer, batch Batch) error
• type Array
  • func NewArray(vals []zed.Value) *Array
  • func ReadAll(r zio.Reader) (arr *Array, err error)
  • func (a *Array) Append(r *zed.Value)
  • func (*Array) CopyValue(val *zed.Value) *zed.Value
  • func (a Array) NewReader() zio.Reader
  • func (*Array) NewValue(typ zed.Type, bytes zcode.Bytes) *zed.Value
  • func (a *Array) Read() (*zed.Value, error)
  • func (a *Array) Ref()
  • func (a *Array) Unref()
  • func (a *Array) Values() []zed.Value
  • func (a *Array) Vars() []zed.Value
  • func (a *Array) Write(r *zed.Value) error
• type Batch
  • func NewBatch(b Batch, vals []zed.Value) Batch
• type Control
  • func (c *Control) Error() string
• type EndChannel
• type File
  • func NewFile(r zio.Reader, c io.Closer, name string) *File
  • func (r *File) Close() error
  • func (r *File) String() string
• type Filter
• type Meter
• type MultiStats
  • func (m MultiStats) Progress() Progress
• type Progress
  • func (p *Progress) Add(in Progress)
  • func (p *Progress) Copy() Progress
  • func (p *Progress) Progress() Progress
• type ProgressReadCloser
  • func MeterReadCloser(rc zio.ReadCloser) ProgressReadCloser
• type ProgressReader
• type Puller
  • func NewPuller(zr zio.Reader) Puller
• type Scanner
  • func MultiScanner(scanners ...Scanner) Scanner
  • func NewScanner(ctx context.Context, r zio.Reader, filterExpr Filter) (Scanner, error)
• type ScannerAble
• type SetChannel

Constants

const PullerBatchBytes = 512 * 1024

PullerBatchBytes is the maximum number of bytes (in the zed.Value.Byte sense) per batch for a Puller created by NewPuller.

Variables

var PullerBatchSize = 100

PullerBatchSize is the maximum number of values per batch for a Puller created by NewPuller.

Functions

func CopyPuller

func CopyPuller(w zio.Writer, p Puller) error

func CopyVars

func CopyVars(b Batch) []zed.Value

func NamedScanner

func NamedScanner(s Scanner, name string) *namedScanner

func NewComparator

func NewComparator(zctx *zed.Context, layout order.Layout) *expr.Comparator

func NewComparatorNullsMax

func NewComparatorNullsMax(zctx *zed.Context, layout order.Layout) *expr.Comparator

func NoControl

func NoControl(r zio.Reader) *noControl

func PullerReader

func PullerReader(p Puller) zio.Reader

func WriteBatch

func WriteBatch(zw zio.Writer, batch Batch) error

WriteBatch writes the values in batch to zw. If an error occurs, WriteBatch stops and returns the error.

Types

type Array

type Array struct {
	// contains filtered or unexported fields
}

Array is a slice of of records that implements the Batch and the Reader interfaces.

func NewArray

func NewArray(vals []zed.Value) *Array

XXX this should take the frame arg too and the procs that create new arrays need to propagate their frames downstream.

func ReadAll

func ReadAll(r zio.Reader) (arr *Array, err error)

func (*Array) Append

func (a *Array) Append(r *zed.Value)

func (*Array) CopyValue

func (*Array) CopyValue(val *zed.Value) *zed.Value

func (Array) NewReader

func (a Array) NewReader() zio.Reader

func (*Array) NewValue

func (*Array) NewValue(typ zed.Type, bytes zcode.Bytes) *zed.Value

func (*Array) Read

func (a *Array) Read() (*zed.Value, error)

Read returns removes the first element of the Array and returns it, or it returns nil if the Array is empty.

func (*Array) Ref

func (a *Array) Ref()

func (*Array) Unref

func (a *Array) Unref()

func (*Array) Values

func (a *Array) Values() []zed.Value

func (*Array) Vars

func (a *Array) Vars() []zed.Value

func (*Array) Write

func (a *Array) Write(r *zed.Value) error

type Batch

type Batch interface {
	expr.Context
	Ref()
	Unref()
	Values() []zed.Value
}

Batch is an interface to a bundle of values. Reference counting allows efficient, safe reuse in concert with sharing across goroutines.

A newly obtained Batch always has a reference count of one. The Batch owns its values and their storage, and an implementation may reuse this memory when the reference count falls to zero, reducing load on the garbage collector.

To promote reuse, a goroutine should release a Batch reference when possible, but care must be taken to avoid race conditions that arise from releasing a reference too soon. Specifically, a goroutine obtaining a value from a Batch must retain its Batch reference for as long as it retains the value, and the goroutine must not use the value after releasing its reference.

Regardless of reference count or implementation, an unreachable Batch will eventually be reclaimed by the garbage collector.

func NewBatch

func NewBatch(b Batch, vals []zed.Value) Batch

type Control

type Control struct {
	Message interface{}
}

func (*Control) Error

func (c *Control) Error() string

type EndChannel

type EndChannel int

type File

type File struct {
	zio.Reader
	// contains filtered or unexported fields
}

func NewFile

func NewFile(r zio.Reader, c io.Closer, name string) *File

func (*File) Close

func (r *File) Close() error

func (*File) String

func (r *File) String() string

type Filter

type Filter interface {
	AsEvaluator() (expr.Evaluator, error)
	AsBufferFilter() (*expr.BufferFilter, error)
}

type Meter

type Meter interface {
	Progress() Progress
}

A Meter provides Progress statistics.

type MultiStats

type MultiStats []Scanner

func (MultiStats) Progress

func (m MultiStats) Progress() Progress

type Progress

type Progress struct {
	BytesRead      int64 `zed:"bytes_read" json:"bytes_read"`
	BytesMatched   int64 `zed:"bytes_matched" json:"bytes_matched"`
	RecordsRead    int64 `zed:"records_read" json:"records_read"`
	RecordsMatched int64 `zed:"records_matched" json:"records_matched"`
}

Progress represents progress statistics from a Scanner.

func (*Progress) Add

func (p *Progress) Add(in Progress)

Add updates its receiver by adding to it the values in ss.

func (*Progress) Copy

func (p *Progress) Copy() Progress

func (*Progress) Progress

func (p *Progress) Progress() Progress

type ProgressReadCloser

type ProgressReadCloser interface {
	zio.ReadCloser
	Progress() Progress
}

func MeterReadCloser

func MeterReadCloser(rc zio.ReadCloser) ProgressReadCloser

type ProgressReader

type ProgressReader interface {
	zio.Reader
	Progress() Progress
}

type Puller

type Puller interface {
	Pull(bool) (Batch, error)
}

A Puller produces Batches of records, signaling end-of-stream (EOS) by returning a nil Batch and nil error. The done argument to Pull indicates that the stream should be terminated before its natural EOS. An implementation must return EOS in response to a Pull call when the done parameter is true. After seeing EOS, (either via done or its natural end), an implementation of an operator that processes pulled data should respond to additional calls to Pull as if restarting in its initial state. For original sources of data (e.g., the from operator), once EOS is reached, additional calls to Pull after the first EOS should always return EOS. Pull is not safe to call concurrently.

func NewPuller

func NewPuller(zr zio.Reader) Puller

NewPuller returns a puller for zr that returns batches containing up to PullerBatchBytes bytes and PullerBatchSize values.

type Scanner

type Scanner interface {
	Meter
	Puller
}

A Scanner is a Batch source that also provides progress updates.

func MultiScanner

func MultiScanner(scanners ...Scanner) Scanner

func NewScanner

func NewScanner(ctx context.Context, r zio.Reader, filterExpr Filter) (Scanner, error)

NewScanner returns a Scanner for r that filters records by filterExpr and s. If r implements fmt.Stringer, the scanner reports errors using a prefix of the string returned by its String method.

type ScannerAble

type ScannerAble interface {
	NewScanner(ctx context.Context, filterExpr Filter) (Scanner, error)
}

ScannerAble is implemented by Readers that provide an optimized implementation of the Scanner interface.

type SetChannel

type SetChannel int