Documentation
¶
Overview ¶
Package vector implements the organization of Zed data on storage as vectors in a ZST vector storage object.
A ZST object is created by allocating a Writer for any top-level Zed type via NewWriter. The object to be written is wrapped in a Spiller with a vector threshold. Output is streamed to the underlying spiller in a single pass.
NewWriter recursively descends into the Zed type, allocating a Writer for each node in the type tree. The top-level body is written via a call to Write. Each vector buffers its data in memory until it reaches a byte threshold or until Flush is called.
After all of the Zed data is written, a metadata section is written consisting of segment maps for each vector, each obtained by calling the Metadata method on the zst.Writer interface.
Nulls for complex types are encoded by a special Nulls object. Each complex type is wrapped by a NullsWriter, which runlength encodes any alternating sequences of nulls and values. If no nulls are encountered, then the Nulls object is omitted from the metadata.
Data is read from a ZST file by scanning the metadata maps to build vector Readers for each Zed type by calling NewReader with the metadata, which recusirvely builds reassembly segments. An io.ReaderAt is passed to NewReader so each vector reader can access the underlying storage object and read its vector data effciently in largish vector segments.
Once the metadata is assembled in memory, the recontructed Zed sequence data can be read from the vector segments by calling the Read method on the top-level Reader and passing in a zcode.Builder to reconstruct the Zed value in place.
Index ¶
- Constants
- Variables
- type Array
- type ArrayReader
- type ArrayWriter
- type Field
- type FieldReader
- type FieldWriter
- type Int64Reader
- type Int64Writer
- type Map
- type MapReader
- type MapWriter
- type Metadata
- type Named
- type NamedWriter
- type Nulls
- type NullsReader
- type NullsWriter
- type Primitive
- type PrimitiveReader
- type PrimitiveWriter
- type Reader
- type Record
- type RecordReader
- type RecordWriter
- type Segment
- type Set
- type SetWriter
- type Spiller
- type Union
- type UnionReader
- type UnionWriter
- type Writer
Constants ¶
const MaxSegmentThresh = 20 * 1024 * 1024
Variables ¶
var ErrVectorMismatch = errors.New("zng record value doesn't match vector writer")
Functions ¶
This section is empty.
Types ¶
type ArrayReader ¶
type ArrayReader struct {
// contains filtered or unexported fields
}
func NewArrayReader ¶
func NewArrayReader(array *Array, r io.ReaderAt) (*ArrayReader, error)
type ArrayWriter ¶
type ArrayWriter struct {
// contains filtered or unexported fields
}
func NewArrayWriter ¶
func NewArrayWriter(inner zed.Type, spiller *Spiller) *ArrayWriter
func (*ArrayWriter) Flush ¶
func (a *ArrayWriter) Flush(eof bool) error
func (*ArrayWriter) Metadata ¶
func (a *ArrayWriter) Metadata() Metadata
type FieldReader ¶
type FieldReader struct {
// contains filtered or unexported fields
}
func NewFieldReader ¶
func NewFieldReader(field Field, r io.ReaderAt) (*FieldReader, error)
type FieldWriter ¶
type FieldWriter struct {
// contains filtered or unexported fields
}
func (*FieldWriter) Flush ¶
func (f *FieldWriter) Flush(eof bool) error
func (*FieldWriter) Metadata ¶
func (f *FieldWriter) Metadata() Field
type Int64Reader ¶
type Int64Reader struct {
PrimitiveReader
}
func NewInt64Reader ¶
func NewInt64Reader(segmap []Segment, r io.ReaderAt) *Int64Reader
func (*Int64Reader) Read ¶
func (p *Int64Reader) Read() (int64, error)
type Int64Writer ¶
type Int64Writer struct {
PrimitiveWriter
}
func NewInt64Writer ¶
func NewInt64Writer(spiller *Spiller) *Int64Writer
func (*Int64Writer) Write ¶
func (p *Int64Writer) Write(v int64) error
type NamedWriter ¶
type NamedWriter struct {
Writer
// contains filtered or unexported fields
}
func (*NamedWriter) Metadata ¶
func (n *NamedWriter) Metadata() Metadata
type NullsReader ¶
type NullsReader struct {
// contains filtered or unexported fields
}
func NewNullsReader ¶
func NewNullsReader(vals Reader, segmap []Segment, r io.ReaderAt) *NullsReader
type NullsWriter ¶
type NullsWriter struct {
// contains filtered or unexported fields
}
NullsWriter emits a sequence of runs of the length of alternating sequences of nulls and values, beginning with nulls. Every run is non-zero except for the first, which may be zero when the first value is non-null.
func NewNullsWriter ¶
func NewNullsWriter(values Writer, spiller *Spiller) *NullsWriter
func (*NullsWriter) Flush ¶
func (n *NullsWriter) Flush(eof bool) error
func (*NullsWriter) Metadata ¶
func (n *NullsWriter) Metadata() Metadata
type PrimitiveReader ¶
type PrimitiveReader struct {
// contains filtered or unexported fields
}
func NewPrimitiveReader ¶
func NewPrimitiveReader(primitive *Primitive, reader io.ReaderAt) *PrimitiveReader
type PrimitiveWriter ¶
type PrimitiveWriter struct {
// contains filtered or unexported fields
}
func NewPrimitiveWriter ¶
func NewPrimitiveWriter(typ zed.Type, spiller *Spiller) *PrimitiveWriter
func (*PrimitiveWriter) Flush ¶
func (p *PrimitiveWriter) Flush(eof bool) error
func (*PrimitiveWriter) Metadata ¶
func (p *PrimitiveWriter) Metadata() Metadata
func (*PrimitiveWriter) Segmap ¶
func (p *PrimitiveWriter) Segmap() []Segment
type RecordReader ¶
type RecordReader []FieldReader
func NewRecordReader ¶
func NewRecordReader(record *Record, reader io.ReaderAt) (RecordReader, error)
type RecordWriter ¶
type RecordWriter []*FieldWriter
func NewRecordWriter ¶
func NewRecordWriter(typ *zed.TypeRecord, spiller *Spiller) RecordWriter
func (RecordWriter) Flush ¶
func (r RecordWriter) Flush(eof bool) error
func (RecordWriter) Metadata ¶
func (r RecordWriter) Metadata() Metadata
type SetWriter ¶
type SetWriter struct {
ArrayWriter
}
type UnionReader ¶
type UnionReader struct {
// contains filtered or unexported fields
}
func NewUnionReader ¶
func NewUnionReader(union *Union, r io.ReaderAt) (*UnionReader, error)
type UnionWriter ¶
type UnionWriter struct {
// contains filtered or unexported fields
}
func NewUnionWriter ¶
func NewUnionWriter(typ *zed.TypeUnion, spiller *Spiller) *UnionWriter
func (*UnionWriter) Flush ¶
func (u *UnionWriter) Flush(eof bool) error
func (*UnionWriter) Metadata ¶
func (u *UnionWriter) Metadata() Metadata
type Writer ¶
type Writer interface {
// Write encodes the given value into memory. When the vector exceeds
// a threshold, it is automatically flushed. Flush may also be called
// explicitly to push vectors to storage and thus avoid too much row skew
// between vectors.
Write(zcode.Bytes) error
// Push all in-memory vector data to the storage layer.
Flush(bool) error
// Metadata returns the data structure conforming to the ZST specification
// describing the layout of vectors. This is called after all data is
// written and flushed by the Writer with the result marshaled to build
// the metadata section of the ZST file.
Metadata() Metadata
}
Source Files