index

Documentation

Overview

Package index provides an API for creating, merging, indexing, and querying Zed indexes.

A Zed index comprises a base index section followed by zero or more parent section indexes followed by a trailer. The sections are organized into a B-tree-like data structure so keys can be looked up efficiently without necessarily scanning the entire base index.

The trailer provides meta information about the index, e.g., indicating the sizes of each section (so section boundaries can be found), the keys that were indexed, the frame threshold used in build the B-tree hierarchy, etc.

Reader implements zio.Reader and Writer implements zio.Writer so generic zng functionality applies, e.g., a Reader can be copied to a Writer using zio.Copy.

Index

• Constants
• Variables
• type FileMeta
• type Finder
  • func NewFinder(ctx context.Context, zctx *zed.Context, engine storage.Engine, ...) (*Finder, error)
  • func (f *Finder) Lookup(kvs ...KeyValue) (*zed.Value, error)
  • func (f *Finder) LookupAll(ctx context.Context, hits chan<- *zed.Value, kvs []KeyValue) error
  • func (f *Finder) Nearest(operator string, kvs ...KeyValue) (*zed.Value, error)
  • func (f *Finder) ParseKeys(inputs ...string) ([]KeyValue, error)
• type FinderReader
  • func NewFinderReader(ctx context.Context, zctx *zed.Context, engine storage.Engine, ...) (*FinderReader, error)
  • func (f *FinderReader) Close() error
  • func (f *FinderReader) Read() (*zed.Value, error)
• type KeyValue
• type Keyer
  • func NewKeyer(zctx *zed.Context, keys []field.Path) (*Keyer, error)
  • func (k *Keyer) Keys() []field.Path
• type Operator
• type Reader
  • func NewReader(zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)
  • func NewReaderFromURI(ctx context.Context, zctx *zed.Context, engine storage.Engine, ...) (*Reader, error)
  • func NewReaderWithContext(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) IsEmpty() bool
  • func (r *Reader) Keys() field.List
  • func (r *Reader) NewSectionReader(section int) (*zngio.Reader, error)
  • func (r *Reader) Order() order.Which
  • func (r *Reader) Path() string
• type Writer
  • func NewWriter(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string, ...) (*Writer, error)
  • func (w *Writer) Abort() error
  • func (w *Writer) Close() error
  • func (w *Writer) Write(val *zed.Value) error
• type WriterOpts

Constants

const (
	FrameFudge   = 1024
	FrameBufSize = frameThresh + FrameFudge
	FrameMaxSize = 20 * 1024 * 1024
)

const (
	FileType       = "index"
	Version        = 4
	ChildFieldName = "_child"
)

const MaxLevels = 20

Variables

var (
	ErrNotIndex        = errors.New("not a Zed index")
	ErrTrailerNotFound = errors.New("Zed index trailer not found")
)

var ErrNotFound = errors.New("key not found")

var (
	ErrTooManyLevels = errors.New("Zed index has too many levels (a larger frame threshold is needed)")
)

Types

type FileMeta

type FileMeta struct {
	Order            order.Which `zed:"order"`
	ChildOffsetField string      `zed:"child_field"`
	FrameThresh      int         `zed:"frame_thresh"`
	Keys             field.List  `zed:"keys"`
}

type Finder

type Finder struct {
	*Reader
	// contains filtered or unexported fields
}

Finder looks up values in a microindex using its embedded index.

func NewFinder

func NewFinder(ctx context.Context, zctx *zed.Context, engine storage.Engine, uri *storage.URI) (*Finder, error)

NewFinder returns an object that is used to lookup keys in a microindex. It opens the file and reads the trailer, returning errors if the file is corrupt, doesn't exist, or has an invalid trailer. If the microindex exists but is empty, zero values are returned for any lookups. If the microindex does not exist, a wrapped zqe.NotFound error is returned.

func (*Finder) Lookup

func (f *Finder) Lookup(kvs ...KeyValue) (*zed.Value, error)

func (*Finder) LookupAll

func (f *Finder) LookupAll(ctx context.Context, hits chan<- *zed.Value, kvs []KeyValue) error

func (*Finder) Nearest

func (f *Finder) Nearest(operator string, kvs ...KeyValue) (*zed.Value, error)

Nearest finds the zed.Value in the index that is nearest to kvs according to operator.

func (*Finder) ParseKeys

func (f *Finder) ParseKeys(inputs ...string) ([]KeyValue, error)

ParseKeys uses the key template from the microindex trailer to parse a slice of string values which correspnod to the DFS-order of the fields in the key. The inputs may be smaller than the number of key fields, in which case they are "don't cares" in terms of key lookups. Any don't-care fields must all be at the end of the key record.

type FinderReader

type FinderReader struct {
	// contains filtered or unexported fields
}

FinderReader is a zio.ReadCloser version of Finder that streams back all records in a microindex that match the provided key record.

func NewFinderReader

func NewFinderReader(ctx context.Context, zctx *zed.Context, engine storage.Engine, uri *storage.URI, inputs ...string) (*FinderReader, error)

NewFinderReader returns a new FinderReader for the microindex at uri and the key record in inputs. See Finder.ParseKeys for an explanation of how the key record is constructed from inputs.

It is the caller's responsibility to call Close on the FinderReader when done.

func (*FinderReader) Close

func (f *FinderReader) Close() error

func (*FinderReader) Read

func (f *FinderReader) Read() (*zed.Value, error)

type KeyValue

type KeyValue struct {
	Key   field.Path
	Value zed.Value
}

type Keyer

type Keyer struct {
	// contains filtered or unexported fields
}

func NewKeyer

func NewKeyer(zctx *zed.Context, keys []field.Path) (*Keyer, error)

func (*Keyer) Keys

func (k *Keyer) Keys() []field.Path

type Operator

type Operator string

const (
	EQL Operator = "=="
	GT  Operator = ">"
	GTE Operator = ">="
	LT  Operator = "<"
	LTE Operator = "<="
)

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader

func NewReader(zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)

NewReader returns a Reader ready to read a microindex. Close() should be called when done. This embeds a zngio.Seeker so Seek() may be called on this Reader. Any call to Seek() must be to an offset that begins a new zng stream (e.g., beginning of file or the data immediately following an end-of-stream code)

func NewReaderFromURI

func NewReaderFromURI(ctx context.Context, zctx *zed.Context, engine storage.Engine, uri *storage.URI) (*Reader, error)

func NewReaderWithContext

func NewReaderWithContext(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)

func (*Reader) Close

func (r *Reader) Close() error

func (*Reader) IsEmpty

func (r *Reader) IsEmpty() bool

func (*Reader) Keys

func (r *Reader) Keys() field.List

func (*Reader) NewSectionReader

func (r *Reader) NewSectionReader(section int) (*zngio.Reader, error)

func (*Reader) Order

func (r *Reader) Order() order.Which

func (*Reader) Path

func (r *Reader) Path() string

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer implements the zio.Writer interface. A Writer creates a Zed index, comprising the base zng file along with its related B-tree sections, as zng records are consumed.

The keyFields argument to NewWriter provides a list of the key names, ordered by precedence, that will serve as the keys into the index. The input records may or may not have all the key fields. If a key field is missing, it appears as a null value in the index. Nulls are sorted before all non-null values. All key fields must have the same type. The Writer may detect an error if a key field changes type but does not check that every key has the same type; it is up to the caller to guarantee this type consistency. For example, the caller should create a separate index for fields that have a common name but different types.

The keys in the input zng stream must be previously sorted consistent with the precedence order of the keyFields.

As the zng file data is written, a B-tree index is computed as a constant B-tree to make key lookups efficient. The B-tree sections are written to temporary files and at close, they are merged into a single-file index.

If a Writer is created but Closed without ever writing records to it, then the index is created with no keys and an "empty" index trailer. This is useful for knowing when something has been indexed but no keys were present. If a Writer is created then an error is enountered (for example, the type of key changes), then you generally want to abort and cleanup by calling Abort() instead of Close().

func NewWriter

func NewWriter(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string, keys field.List,
	opts WriterOpts) (*Writer, error)

NewWriter returns a Writer ready to write a Zed index or it returns an error. The index is written to the URL provided in the path argument while temporary file are written locally. Calls to Write must provide keys in increasing lexicographic order. Duplicate keys are not allowed but will not be detected. Close() or Abort() must be called when done writing.

func (*Writer) Abort

func (w *Writer) Abort() error

Abort closes this writer, deleting any and all objects and/or files associated with it.

func (*Writer) Close

func (w *Writer) Close() error

func (*Writer) Write

func (w *Writer) Write(val *zed.Value) error

type WriterOpts

type WriterOpts struct {
	FrameThresh   int
	Order         order.Which
	ZNGWriterOpts *zngio.WriterOpts
}