zst

Documentation

Overview

Package zst implements the reading and writing of ZST storage objects to and from any Zed format. The ZST storage format is described at https://github.com/brimdata/zed/blob/main/docs/formats/zst.md.

A ZST storage object must be seekable (e.g., a local file or S3 object), so, unlike ZNG, streaming of ZST objects is not supported.

The zst/column package handles reading and writing row data to columns, while the zst package comprises the API used to read and write ZST objects.

An Object provides the interface to the underlying storage object. To generate rows or cuts (and in the future more sophisticated traversals and introspection), an Assembly is created from the Object then zng records are read from the assembly, which implements zio.Reader. The Assembly keeps track of where each column is, which is why you need a separate Assembly per scan.

You can have multiple Assembly's referring to one Object as once an object is created, it's state never changes. That said, each assembly will issue reads to the underlying storage object and the read pattern may create performance issues.

Index

• Constants
• Variables
• type Assembler
  • func NewAssembler(a *Assembly, seeker *storage.Seeker) (*Assembler, error)
  • func (a *Assembler) Read() (*zed.Value, error)
• type Assembly
• type CutAssembler
  • func NewCutAssembler(zctx *zed.Context, fields []string, object *Object) (*CutAssembler, error)
  • func (a *CutAssembler) Read() (*zed.Value, error)
• type FileMeta
• type Object
  • func NewObject(zctx *zed.Context, s *storage.Seeker, size int64) (*Object, error)
  • func NewObjectFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Object, error)
  • func NewObjectFromSeeker(zctx *zed.Context, s *storage.Seeker) (*Object, error)
  • func (o *Object) Close() error
  • func (o *Object) IsEmpty() bool
  • func (o *Object) NewReassemblyReader() zio.Reader
• type Reader
  • func NewCutter(object *Object, fields []string) (*Reader, error)
  • func NewCutterFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string, ...) (*Reader, error)
  • func NewReader(object *Object) (*Reader, error)
  • func NewReaderFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)
  • func NewReaderFromSeeker(zctx *zed.Context, seeker *storage.Seeker) (*Reader, error)
• type Writer
  • func NewWriter(w io.WriteCloser, skewThresh, segThresh int) (*Writer, error)
  • func (w *Writer) Close() error
  • func (w *Writer) Write(val *zed.Value) error
• type WriterURI
  • func NewWriterFromPath(ctx context.Context, engine storage.Engine, path string, ...) (*WriterURI, error)
  • func (w *WriterURI) Abort(ctx context.Context) error
  • func (w *WriterURI) Close() error

Constants

const (
	FileType = "zst"
	Version  = 2
)

const (
	MaxSegmentThresh = column.MaxSegmentThresh
	MaxSkewThresh    = 512 * 1024 * 1024
)

Variables

var ErrBadTypeNumber = errors.New("bad type number in ZST root reassembly map")

Types

type Assembler

type Assembler struct {
	// contains filtered or unexported fields
}

Assembler reads a columnar ZST object to generate a stream of zed.Values. It also has methods to read metadata for test and debugging.

func NewAssembler

func NewAssembler(a *Assembly, seeker *storage.Seeker) (*Assembler, error)

func (*Assembler) Read

func (a *Assembler) Read() (*zed.Value, error)

type Assembly

type Assembly struct {
	// contains filtered or unexported fields
}

type CutAssembler

type CutAssembler struct {
	// contains filtered or unexported fields
}

func NewCutAssembler

func NewCutAssembler(zctx *zed.Context, fields []string, object *Object) (*CutAssembler, error)

func (*CutAssembler) Read

func (a *CutAssembler) Read() (*zed.Value, error)

type FileMeta

type FileMeta struct {
	SkewThresh    int `zed:"skew_thresh"`
	SegmentThresh int `zed:"segment_thresh"`
}

type Object

type Object struct {
	// contains filtered or unexported fields
}

func NewObject

func NewObject(zctx *zed.Context, s *storage.Seeker, size int64) (*Object, error)

func NewObjectFromPath

func NewObjectFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Object, error)

func NewObjectFromSeeker

func NewObjectFromSeeker(zctx *zed.Context, s *storage.Seeker) (*Object, error)

func (*Object) Close

func (o *Object) Close() error

func (*Object) IsEmpty

func (o *Object) IsEmpty() bool

func (*Object) NewReassemblyReader

func (o *Object) NewReassemblyReader() zio.Reader

type Reader

type Reader struct {
	*Object
	zio.Reader
}

Reader implements zio.Reader and io.Closer. It reads a columnar zst object to generate a stream of zed.Records. It also has methods to read metainformation for test and debugging.

func NewCutter

func NewCutter(object *Object, fields []string) (*Reader, error)

func NewCutterFromPath

func NewCutterFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string, fields []string) (*Reader, error)

func NewReader

func NewReader(object *Object) (*Reader, error)

NewReader returns a Reader ready to read a zst object as zed.Records. Close() should be called when done. This embeds a zst.Object.

func NewReaderFromPath

func NewReaderFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)

func NewReaderFromSeeker

func NewReaderFromSeeker(zctx *zed.Context, seeker *storage.Seeker) (*Reader, error)

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer implements the zio.Writer interface. A Writer creates a columnar zst object from a stream of zed.Records.

func NewWriter

func NewWriter(w io.WriteCloser, skewThresh, segThresh int) (*Writer, error)

func (*Writer) Close

func (w *Writer) Close() error

func (*Writer) Write

func (w *Writer) Write(val *zed.Value) error

type WriterURI

type WriterURI struct {
	Writer
	// contains filtered or unexported fields
}

func NewWriterFromPath

func NewWriterFromPath(ctx context.Context, engine storage.Engine, path string, skewThresh, segThresh int) (*WriterURI, error)

func (*WriterURI) Abort

func (w *WriterURI) Abort(ctx context.Context) error

Abort closes this writer, deleting any and all objects and/or files associated with it.

func (*WriterURI) Close

func (w *WriterURI) Close() error