Documentation
¶
Overview ¶
Package column implements the organization of columns on storage for a ZST columnar storage object.
A ZST object is created by allocating a Writer for any top-level Zed type via NewWriter. The object to be written is wrapped in a Spiller with a column threshold. Output is streamed to the underlying spiller in a single pass. (In the future, we may implement multiple passes to optimize the storage layout of column data or spread a given ZST object across multiple files.
NewWriter recursively decends into the Zed type, allocating a Writer for each node in the type tree. The top-level body is written via a call to Write. The columns buffer data in memory until they reach their byte threshold or until Flush is called.
After all of the Zed data is written, a reassembly map is formed for each column writer by calling its EncodeMap method, which builds the value in place using zcode.Builder and returns the Zed type of the reassembly map value.
Data is read from a ZST file by scanning the reassembly maps to build column Readers for each Zed type by calling NewReader with the map, which recusirvely builds an assembly structure. An io.ReaderAt is passed to NewReader so each column reader can access the underlying storage object and read its column data effciently in largish column chunks.
Once an assembly is built, the recontructed Zed row data can be read from the assembly by calling the Read method on the top-level Record and passing in a zcode.Builder to reconstruct the record body in place. The assembly does not need any type information as the structure of values is entirely self describing in the Zed data format.
Index ¶
- Constants
- Variables
- type ArrayReader
- type ArrayWriter
- type FieldReader
- type FieldWriter
- type IntReader
- type IntWriter
- type PresenceReader
- type PresenceWriter
- type PrimitiveReader
- type PrimitiveWriter
- type Reader
- type RecordReader
- type RecordWriter
- type Segment
- type Spiller
- type UnionReader
- type UnionWriter
- type Writer
Constants ¶
const MaxSegmentThresh = 20 * 1024 * 1024
const SegmapTypeString = "[{offset:int64,length:int32}]"
Variables ¶
var ErrColumnMismatch = errors.New("zng record value doesn't match column writer")
var ErrCorruptSegment = errors.New("segmap value corrupt")
var ErrNonRecordAccess = errors.New("attempting to access a field in a non-record value")
Functions ¶
This section is empty.
Types ¶
type ArrayReader ¶ added in v1.0.0
type ArrayReader struct {
// contains filtered or unexported fields
}
func NewArrayReader ¶ added in v1.0.0
type ArrayWriter ¶
type ArrayWriter struct {
// contains filtered or unexported fields
}
func NewArrayWriter ¶
func NewArrayWriter(inner zed.Type, spiller *Spiller) *ArrayWriter
func (*ArrayWriter) Flush ¶
func (a *ArrayWriter) Flush(eof bool) error
type FieldReader ¶ added in v1.0.0
type FieldReader struct {
// contains filtered or unexported fields
}
func NewFieldReader ¶ added in v1.0.0
type FieldWriter ¶
type FieldWriter struct {
// contains filtered or unexported fields
}
func (*FieldWriter) Flush ¶
func (f *FieldWriter) Flush(eof bool) error
type IntReader ¶ added in v1.0.0
type IntReader struct {
PrimitiveReader
}
func NewIntReader ¶ added in v1.0.0
type PresenceReader ¶ added in v1.0.0
type PresenceReader struct {
IntReader
// contains filtered or unexported fields
}
func NewPresence ¶
func NewPresence(i IntReader) *PresenceReader
func (*PresenceReader) IsEmpty ¶ added in v1.0.0
func (p *PresenceReader) IsEmpty() bool
func (*PresenceReader) Read ¶ added in v1.0.0
func (p *PresenceReader) Read() (bool, error)
type PresenceWriter ¶
type PresenceWriter struct {
IntWriter
// contains filtered or unexported fields
}
func NewPresenceWriter ¶
func NewPresenceWriter(spiller *Spiller) *PresenceWriter
func (*PresenceWriter) Finish ¶
func (p *PresenceWriter) Finish()
func (*PresenceWriter) TouchNull ¶ added in v1.0.0
func (p *PresenceWriter) TouchNull()
func (*PresenceWriter) TouchValue ¶
func (p *PresenceWriter) TouchValue()
type PrimitiveReader ¶ added in v1.0.0
type PrimitiveReader struct {
// contains filtered or unexported fields
}
func NewPrimitiveReader ¶ added in v1.0.0
type PrimitiveWriter ¶
type PrimitiveWriter struct {
// contains filtered or unexported fields
}
func NewPrimitiveWriter ¶
func NewPrimitiveWriter(spiller *Spiller) *PrimitiveWriter
func (*PrimitiveWriter) Flush ¶
func (p *PrimitiveWriter) Flush(eof bool) error
type RecordReader ¶ added in v1.0.0
type RecordReader []FieldReader
func NewRecordReader ¶ added in v1.0.0
func (RecordReader) Lookup ¶ added in v1.0.0
func (r RecordReader) Lookup(typ *zed.TypeRecord, fields []string) (zed.Type, Reader, error)
type RecordWriter ¶
type RecordWriter []*FieldWriter
func NewRecordWriter ¶
func NewRecordWriter(typ *zed.TypeRecord, spiller *Spiller) RecordWriter
func (RecordWriter) Flush ¶
func (r RecordWriter) Flush(eof bool) error
type Segment ¶
func NewSegment ¶ added in v1.0.0
type UnionReader ¶ added in v1.0.0
type UnionReader struct {
// contains filtered or unexported fields
}
func NewUnionReader ¶ added in v1.0.0
type UnionWriter ¶
type UnionWriter struct {
// contains filtered or unexported fields
}
func NewUnionWriter ¶
func NewUnionWriter(typ *zed.TypeUnion, spiller *Spiller) *UnionWriter
func (*UnionWriter) Flush ¶
func (u *UnionWriter) Flush(eof bool) error
type Writer ¶
type Writer interface {
// Write encodes the given value into memory. When the column exceeds
// a threshold, it is automatically flushed. Flush may also be called
// explicitly to push columns to storage and thus avoid too much row skew
// between columns.
Write(zcode.Bytes) error
// Push all in-memory column data to the storage layer.
Flush(bool) error
// EncodeMap is called after all data is flushed to build the reassembly
// record for this column.
EncodeMap(*zed.Context, *zcode.Builder) (zed.Type, error)
}
Source Files