zbuf

Documentation

Index

• Constants
• Variables
• func CopyMux(outputs map[string]zio.WriteCloser, parent Puller) error
• func CopyPuller(w zio.Writer, p Puller) error
• func CopyVars(b Batch) []zed.Value
• func NamedScanner(s Scanner, name string) *namedScanner
• func NewComparator(zctx *zed.Context, sortKeys []order.SortKey) *expr.Comparator
• func NewComparatorNullsMax(zctx *zed.Context, sortKeys order.SortKeys) *expr.Comparator
• func PullerReader(p Puller) zio.Reader
• func WriteBatch(zw zio.Writer, batch Batch) error
• type Array
  • func NewArray(vals []zed.Value) *Array
  • func (a *Array) Append(r zed.Value)
  • func (a *Array) Read() (*zed.Value, error)
  • func (a *Array) Ref()
  • func (a *Array) SetVars(vars []zed.Value)
  • func (a *Array) Unref()
  • func (a *Array) Values() []zed.Value
  • func (a *Array) Vars() []zed.Value
  • func (a *Array) Write(r zed.Value) error
• type Batch
  • func Label(label string, batch Batch) Batch
  • func NewBatch(b Batch, vals []zed.Value) Batch
  • func Unlabel(batch Batch) (Batch, string)
• type Control
  • func (c *Control) Error() string
• type EndOfChannel
  • func (*EndOfChannel) Ref()
  • func (*EndOfChannel) Unref()
  • func (*EndOfChannel) Values() []zed.Value
  • func (*EndOfChannel) Vars() []zed.Value
• type File
  • func NewFile(r zio.Reader, c io.Closer, name string) *File
  • func (r *File) Close() error
  • func (r *File) String() string
• type Filter
• type Meter
• type MultiStats
  • func (m MultiStats) Progress() Progress
• type Progress
  • func (p *Progress) Add(in Progress)
  • func (p *Progress) Copy() Progress
  • func (p *Progress) Progress() Progress
• type Puller
  • func NewPuller(zr zio.Reader) Puller
• type Scanner
  • func MultiScanner(scanners ...Scanner) Scanner
  • func NewScanner(ctx context.Context, r zio.Reader, filterExpr Filter) (Scanner, error)
• type ScannerAble

Constants

const PullerBatchBytes = 512 * 1024

PullerBatchBytes is the maximum number of bytes (in the zed.Value.Byte sense) per batch for a Puller created by NewPuller.

Variables

var PullerBatchValues = 100

PullerBatchValues is the maximum number of values per batch for a Puller created by NewPuller.

Functions

func CopyMux

func CopyMux(outputs map[string]zio.WriteCloser, parent Puller) error

func CopyPuller

func CopyPuller(w zio.Writer, p Puller) error

func CopyVars

func CopyVars(b Batch) []zed.Value

func NamedScanner

func NamedScanner(s Scanner, name string) *namedScanner

func NewComparator

func NewComparator(zctx *zed.Context, sortKeys []order.SortKey) *expr.Comparator

func NewComparatorNullsMax

func NewComparatorNullsMax(zctx *zed.Context, sortKeys order.SortKeys) *expr.Comparator

func PullerReader

func PullerReader(p Puller) zio.Reader

func WriteBatch

func WriteBatch(zw zio.Writer, batch Batch) error

WriteBatch writes the values in batch to zw. If an error occurs, WriteBatch stops and returns the error.

Types

type Array

type Array struct {
	// contains filtered or unexported fields
}

Array is a slice of of records that implements the Batch and the Reader interfaces.

func NewArray

func NewArray(vals []zed.Value) *Array

XXX this should take the frame arg too and the procs that create new arrays need to propagate their frames downstream.

func (*Array) Append

func (a *Array) Append(r zed.Value)

func (*Array) Read

func (a *Array) Read() (*zed.Value, error)

Read returns removes the first element of the Array and returns it, or it returns nil if the Array is empty.

func (*Array) Ref

func (a *Array) Ref()

func (*Array) SetVars

func (a *Array) SetVars(vars []zed.Value)

func (*Array) Unref

func (a *Array) Unref()

func (*Array) Values

func (a *Array) Values() []zed.Value

func (*Array) Vars

func (a *Array) Vars() []zed.Value

func (*Array) Write

func (a *Array) Write(r zed.Value) error

type Batch

type Batch interface {
	Ref()
	Unref()
	Values() []zed.Value
	// Vars accesses the variables reachable in the current scope.
	Vars() []zed.Value
}

Batch is an interface to a bundle of values. Reference counting allows efficient, safe reuse in concert with sharing across goroutines.

A newly obtained Batch always has a reference count of one. The Batch owns its values and their storage, and an implementation may reuse this memory when the reference count falls to zero, reducing load on the garbage collector.

To promote reuse, a goroutine should release a Batch reference when possible, but care must be taken to avoid race conditions that arise from releasing a reference too soon. Specifically, a goroutine obtaining a value from a Batch must retain its Batch reference for as long as it retains the value, and the goroutine must not use the value after releasing its reference.

Regardless of reference count or implementation, an unreachable Batch will eventually be reclaimed by the garbage collector.

func Label

func Label(label string, batch Batch) Batch

func NewBatch

func NewBatch(b Batch, vals []zed.Value) Batch

func Unlabel

func Unlabel(batch Batch) (Batch, string)

type Control

type Control struct {
	Message interface{}
}

func (*Control) Error

func (c *Control) Error() string

type EndOfChannel

type EndOfChannel string

EndOfChannel is an empty batch that represents the termination of one of the output paths of a muxed flowgraph and thus will be ignored downstream unless explicitly detected.

func (*EndOfChannel) Ref

func (*EndOfChannel) Ref()

func (*EndOfChannel) Unref

func (*EndOfChannel) Unref()

func (*EndOfChannel) Values

func (*EndOfChannel) Values() []zed.Value

func (*EndOfChannel) Vars

func (*EndOfChannel) Vars() []zed.Value

type File

type File struct {
	zio.Reader
	// contains filtered or unexported fields
}

func NewFile

func NewFile(r zio.Reader, c io.Closer, name string) *File

func (*File) Close

func (r *File) Close() error

func (*File) String

func (r *File) String() string

type Filter

type Filter interface {
	AsEvaluator() (expr.Evaluator, error)
	AsBufferFilter() (*expr.BufferFilter, error)
}

type Meter

type Meter interface {
	Progress() Progress
}

A Meter provides Progress statistics.

type MultiStats

type MultiStats []Scanner

func (MultiStats) Progress

func (m MultiStats) Progress() Progress

type Progress

type Progress struct {
	BytesRead      int64 `zed:"bytes_read" json:"bytes_read"`
	BytesMatched   int64 `zed:"bytes_matched" json:"bytes_matched"`
	RecordsRead    int64 `zed:"records_read" json:"records_read"`
	RecordsMatched int64 `zed:"records_matched" json:"records_matched"`
}

Progress represents progress statistics from a Scanner.

func (*Progress) Add

func (p *Progress) Add(in Progress)

Add updates its receiver by adding to it the values in ss.

func (*Progress) Copy

func (p *Progress) Copy() Progress

func (*Progress) Progress

func (p *Progress) Progress() Progress

type Puller

type Puller interface {
	Pull(bool) (Batch, error)
}

A Puller produces Batches of records, signaling end-of-stream (EOS) by returning a nil Batch and nil error. The done argument to Pull indicates that the stream should be terminated before its natural EOS. An implementation must return EOS in response to a Pull call when the done parameter is true. After seeing EOS, (either via done or its natural end), an implementation of an operator that processes pulled data should respond to additional calls to Pull as if restarting in its initial state. For original sources of data (e.g., the from operator), once EOS is reached, additional calls to Pull after the first EOS should always return EOS. Pull is not safe to call concurrently.

func NewPuller

func NewPuller(zr zio.Reader) Puller

NewPuller returns a puller for zr that returns batches containing up to PullerBatchBytes bytes and PullerBatchValues values.

type Scanner

type Scanner interface {
	Meter
	Puller
}

A Scanner is a Batch source that also provides progress updates.

func MultiScanner

func MultiScanner(scanners ...Scanner) Scanner

func NewScanner

func NewScanner(ctx context.Context, r zio.Reader, filterExpr Filter) (Scanner, error)

NewScanner returns a Scanner for r that filters records by filterExpr and s. If r implements fmt.Stringer, the scanner reports errors using a prefix of the string returned by its String method.

type ScannerAble

type ScannerAble interface {
	NewScanner(ctx context.Context, filterExpr Filter) (Scanner, error)
}

ScannerAble is implemented by Readers that provide an optimized implementation of the Scanner interface.