Documentation ¶
Overview ¶
Package index provides an API for creating, merging, indexing, and querying Zed indexes.
A Zed index comprises a base index section followed by zero or more parent section indexes followed by a trailer. The sections are organized into a B-tree-like data structure so keys can be looked up efficiently without necessarily scanning the entire base index.
The trailer provides meta information about the index, e.g., indicating the sizes of each section (so section boundaries can be found), the keys that were indexed, the frame threshold used in build the B-tree hierarchy, etc.
Reader implements zio.Reader and Writer implements zio.Writer so generic zng functionality applies, e.g., a Reader can be copied to a Writer using zio.Copy.
Index ¶
- Constants
- Variables
- type FileMeta
- type Finder
- type FinderReader
- type KeyValue
- type Keyer
- type Operator
- type Option
- type Reader
- func NewReader(zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)
- func NewReaderFromURI(ctx context.Context, zctx *zed.Context, engine storage.Engine, ...) (*Reader, error)
- func NewReaderWithContext(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)
- type Writer
Constants ¶
const ( FrameFudge = 1024 FrameBufSize = frameThresh + FrameFudge FrameMaxSize = 20 * 1024 * 1024 )
const ( FileType = "index" Version = 4 ChildFieldName = "_child" )
const MaxLevels = 20
Variables ¶
var ( ErrNotIndex = errors.New("not a Zed index") ErrTrailerNotFound = errors.New("Zed index trailer not found") )
var ErrNotFound = errors.New("key not found")
var (
ErrTooManyLevels = errors.New("Zed index has too many levels (a larger frame threshold is needed)")
)
Functions ¶
This section is empty.
Types ¶
type Finder ¶
type Finder struct { *Reader // contains filtered or unexported fields }
Finder looks up values in a microindex using its embedded index.
func NewFinder ¶
func NewFinder(ctx context.Context, zctx *zed.Context, engine storage.Engine, uri *storage.URI) (*Finder, error)
NewFinder returns an object that is used to lookup keys in a microindex. It opens the file and reads the trailer, returning errors if the file is corrupt, doesn't exist, or has an invalid trailer. If the microindex exists but is empty, zero values are returned for any lookups. If the microindex does not exist, a wrapped zqe.NotFound error is returned.
func (*Finder) Nearest ¶ added in v0.32.0
Nearest finds the zed.Value in the index that is nearest to kvs according to operator.
func (*Finder) ParseKeys ¶
ParseKeys uses the key template from the microindex trailer to parse a slice of string values which correspnod to the DFS-order of the fields in the key. The inputs may be smaller than the number of key fields, in which case they are "don't cares" in terms of key lookups. Any don't-care fields must all be at the end of the key record.
type FinderReader ¶
type FinderReader struct {
// contains filtered or unexported fields
}
FinderReader is zio.Reader version of Finder that streams back all records in a microindex that match the provided key Record.
func NewFinderReader ¶
func (*FinderReader) Close ¶
func (f *FinderReader) Close() error
type Option ¶
type Option interface {
// contains filtered or unexported methods
}
func FrameThresh ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
func NewReader ¶
NewReader returns a Reader ready to read a microindex. Close() should be called when done. This embeds a zngio.Seeker so Seek() may be called on this Reader. Any call to Seek() must be to an offset that begins a new zng stream (e.g., beginning of file or the data immediately following an end-of-stream code)
func NewReaderFromURI ¶
func NewReaderWithContext ¶
func (*Reader) NewSectionReader ¶
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer implements the zio.Writer interface. A Writer creates a Zed index, comprising the base zng file along with its related B-tree sections, as zng records are consumed.
The keyFields argument to NewWriter provides a list of the key names, ordered by precedence, that will serve as the keys into the index. The input records may or may not have all the key fields. If a key field is missing, it appears as a null value in the index. Nulls are sorted before all non-null values. All key fields must have the same type. The Writer may detect an error if a key field changes type but does not check that every key has the same type; it is up to the caller to guarantee this type consistency. For example, the caller should create a separate index for fields that have a common name but different types.
The keys in the input zng stream must be previously sorted consistent with the precedence order of the keyFields.
As the zng file data is written, a B-tree index is computed as a constant B-tree to make key lookups efficient. The B-tree sections are written to temporary files and at close, they are merged into a single-file index.
If a Writer is created but Closed without ever writing records to it, then the index is created with no keys and an "empty" index trailer. This is useful for knowing when something has been indexed but no keys were present. If a Writer is created then an error is enountered (for example, the type of key changes), then you generally want to abort and cleanup by calling Abort() instead of Close().
func NewWriter ¶
func NewWriter(zctx *zed.Context, engine storage.Engine, path string, keys field.List, options ...Option) (*Writer, error)
NewWriter returns a Writer ready to write a Zed index or it returns an error. The index is written to the URL provided in the path argument while temporary file are written locally. Calls to Write must provide keys in increasing lexicographic order. Duplicate keys are not allowed but will not be detected. Close() or Abort() must be called when done writing.