zed

package module
v0.34.0-test1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2022 License: BSD-3-Clause Imports: 17 Imported by: 21

README ¶

Zed Tests GoPkg

Zed is a system for search, analytics, and data transformation designed to work at any scale, from a simple command-line tool called zq on your desktop (kind of like jq) to a large-scale, distributed cluster running on your servers or in the cloud (like a search cluster or a data warehouse).

Zed is based on a new type of data called super-structured data, which can represent both relational tables and JSON. Super-structured data provides a new and easier approach for data introspection, shaping, pipeline management, data quality, test, and debug, all while intermixing the flexibility of the JSON document model with the efficiency of warehouse analytics.

For a non-technical user, Zed is as easy to use as web search while for a technical user, Zed exposes its technical underpinnings in a gradual slope, providing as much detail as desired, packaged up in the easy-to-understand ZSON data format and the Zed Lake API.

Check out the Zed FAQ.

Why?

We think that you shouldn't have to set up one system for search and another completely different system for historical analytics. And the same search/analytics system that works at cloud scale should run easily as a lightweight command-line tool on your laptop.

And rather than having to set up complex ETL pipelines with brittle transformation logic, managing your data lake should be as easy as git.

And while schemas are a great way to model and organize your data, they often get in the way when you are just trying to store or transmit your semi-structured data.

Finally, we believe a lightweight data store that provides easy search and analytics would be a great place to store data sets for data science and data engineering experiments running in Python and providing easy integration with your favorite Python libraries.

How?

Zed solves all these problems with a new format called ZSON, which is a superset of JSON and the relational models. ZSON is syntax-compatible with JSON but it has a comprehensive type system that you can use as little or as much as you like. Zed types can be used as schemas.

Zed also has a cloud-based object design that was modeled after the git design pattern. Commits to the lake are transactional and consistent. Search index updates are also transactionally consistent with any ingested data, and searches can run with or without indexes.

Quick Start

Detailed documentation is available.

The quickest way to get running on macOS, Linux, or Windows is to download a pre-built release binary. You can find these binaries on the GitHub releases page.

If you have Go installed, you can easily install zed and zq by running

go install github.com/brimdata/zed/cmd/{zed,zq}@latest

On a Mac, you can use brew to install both zq and zed:

brew install brimdata/zed/zed

If you want just zq, use

brew install brimdata/zed/zq

Once installed, you can run the query engine from the command-line using zq:

echo '"hello, world"' | zq -

Or you can run a Zed lake service, load it with data using zed load, and hit the API. In one shell, run the server:

mkdir scratch
zed serve -lake scratch

And in another shell, run the client:

zed create Demo
zed use Demo@main
echo '{s:"hello, world"}' | zed load -
zed query "from Demo"

You can also use zed from Python. After you install the Zed Python:

pip3 install "git+https://github.com/brimdata/zed#subdirectory=python/zed"

You can hit the Zed service from a Python program:

import zed

# Connect to the default lake at http://localhost:9867.  To use a
# different lake, supply its URL via the ZED_LAKE environment variable
# or as an argument here.
client = zed.Client()

# Begin executing a Zed query for all records in the pool named "Demo".
# This returns an iterator, not a container.
records = client.query('from Demo')

# Stream records from the server.
for record in records:
    print(record)

See the python/zed for more details.

Brim

You can use the Brim app to explore, query, and shape the data in your Zed lake.

We originally developed Brim for security-oriented use cases (having tight integration with Zeek, Suricata, and Wireshark), but we are actively extending Brim with UX for handling generic data sets to support data science, data engineering, and ETL use cases.

Building from Source

It's also easy to build zed from source:

git clone https://github.com/brimdata/zed
cd zed
make install

This installs binaries in your $GOPATH/bin.

If you don't have Go installed, download and install it from the Go install page. Go version 1.16 or later is required.

Contributing

See the contributing guide on how you can help improve Zed!

Join the Community

Join our Public Slack workspace for announcements, Q&A, and to trade tips!

Acknowledgment

We modeled this README after Philip O'Toole's brilliantly succinct description of rqlite.

Documentation ¶

Overview ¶

Package zng implements a data typing system based on the zeek type system. All zeek types are defined here and implement the Type interface while instances of values implement the Value interface. All values conform to exactly one type. The package provides a fast-path for comparing a value to a byte slice without having to create a zeek value from the byte slice. To exploit this, all values include a Comparison method that returns a Predicate function that takes a byte slice and a Type and returns a boolean indicating whether the the byte slice with the indicated Type matches the value. The package also provides mechanism for coercing values in well-defined and natural ways.

Index ¶

Constants ¶

View Source 
const (
	MaxColumns     = 100_000
	MaxEnumSymbols = 100_000
	MaxUnionTypes  = 100_000
)
View Source 
const (
	IDUint8       = 0
	IDUint16      = 1
	IDUint32      = 2
	IDUint64      = 3
	IDUint128     = 4
	IDUint256     = 5
	IDInt8        = 6
	IDInt16       = 7
	IDInt32       = 8
	IDInt64       = 9
	IDInt128      = 10
	IDInt256      = 11
	IDDuration    = 12
	IDTime        = 13
	IDFloat16     = 14
	IDFloat32     = 15
	IDFloat64     = 16
	IDFloat128    = 17
	IDFloat256    = 18
	IDDecimal32   = 19
	IDDecimal64   = 20
	IDDecimal128  = 21
	IDDecimal256  = 22
	IDBool        = 23
	IDBytes       = 24
	IDString      = 25
	IDIP          = 26
	IDNet         = 27
	IDType        = 28
	IDNull        = 29
	IDTypeComplex = 30
)
View Source 
const (
	TypeValueRecord  = 30
	TypeValueArray   = 31
	TypeValueSet     = 32
	TypeValueMap     = 33
	TypeValueUnion   = 34
	TypeValueEnum    = 35
	TypeValueError   = 36
	TypeValueNameDef = 37
	TypeValueNameRef = 38
	TypeValueMax     = TypeValueNameRef
)

Variables ¶

View Source 
var (
	ErrNotArray      = errors.New("cannot index a non-array")
	ErrIndex         = errors.New("array index out of bounds")
	ErrUnionSelector = errors.New("union selector out of bounds")
	ErrEnumIndex     = errors.New("enum index out of bounds")
)
View Source 
var (
	TypeUint8    = &TypeOfUint8{}
	TypeUint16   = &TypeOfUint16{}
	TypeUint32   = &TypeOfUint32{}
	TypeUint64   = &TypeOfUint64{}
	TypeInt8     = &TypeOfInt8{}
	TypeInt16    = &TypeOfInt16{}
	TypeInt32    = &TypeOfInt32{}
	TypeInt64    = &TypeOfInt64{}
	TypeDuration = &TypeOfDuration{}
	TypeTime     = &TypeOfTime{}
	// XXX add TypeFloat16
	TypeFloat32 = &TypeOfFloat32{}
	TypeFloat64 = &TypeOfFloat64{}
	// XXX add TypeDecimal
	TypeBool   = &TypeOfBool{}
	TypeBytes  = &TypeOfBytes{}
	TypeString = &TypeOfString{}
	TypeIP     = &TypeOfIP{}
	TypeNet    = &TypeOfNet{}
	TypeType   = &TypeOfType{}
	TypeNull   = &TypeOfNull{}
)
View Source 
var (
	ErrMissingField  = errors.New("record missing a field")
	ErrExtraField    = errors.New("record with extra field")
	ErrNotContainer  = errors.New("expected container type, got primitive")
	ErrNotPrimitive  = errors.New("expected primitive type, got container")
	ErrTypeIDInvalid = errors.New("zng type ID out of range")
	ErrBadValue      = errors.New("malformed zng value")
	ErrBadFormat     = errors.New("malformed zng record")
	ErrTypeMismatch  = errors.New("type/value mismatch")
	ErrTypeSyntax    = errors.New("syntax error parsing type string")
)
View Source 
var (
	NullUint8    = &Value{Type: TypeUint8}
	NullUint16   = &Value{Type: TypeUint16}
	NullUint32   = &Value{Type: TypeUint32}
	NullUint64   = &Value{Type: TypeUint64}
	NullInt8     = &Value{Type: TypeInt8}
	NullInt16    = &Value{Type: TypeInt16}
	NullInt32    = &Value{Type: TypeInt32}
	NullInt64    = &Value{Type: TypeInt64}
	NullDuration = &Value{Type: TypeDuration}
	NullTime     = &Value{Type: TypeTime}
	NullFloat32  = &Value{Type: TypeFloat32}
	NullFloat64  = &Value{Type: TypeFloat64}
	NullBool     = &Value{Type: TypeBool}
	NullBytes    = &Value{Type: TypeBytes}
	NullString   = &Value{Type: TypeString}
	NullIP       = &Value{Type: TypeIP}
	NullNet      = &Value{Type: TypeNet}
	NullType     = &Value{Type: TypeType}
	Null         = &Value{Type: TypeNull}
)
View Source 
var ErrDuplicateFields = errors.New("duplicate fields")
View Source 
var ErrMissing = errors.New("missing")

ErrMissing is a Go error that implies a missing value in the runtime logic whereas Missing is a Zed error value that represents a missing value embedded in the dataflow computation.

View Source 
var ErrNonAdjacent = errors.New("non adjacent fields")
View Source 
var False = &Value{TypeBool, []byte{0}}
View Source 
var Missing = zcode.Bytes("missing")

Missing is value that represents an error condition arising from a referenced entity not present, e.g., a reference to a non-existent record field, a map lookup for a key not present, an array index that is out of range, etc. The Missing error can be propagated through functions and expressions and each operator has clearly defined semantics with respect to the Missing value. For example, "true AND MISSING" is MISSING.

View Source 
var Quiet = zcode.Bytes("quiet")
View Source 
var SkipContainer = errors.New("skip this container")

SkipContainer is used as a return value from Visitors to indicate that the container passed in the call should not be visited. It is not returned as an error by any function.

View Source 
var True = &Value{TypeBool, []byte{1}}

Functions ¶

func AppendBool ¶ 

func AppendBool(zb zcode.Bytes, b bool) zcode.Bytes

func AppendDuration ¶ 

func AppendDuration(bytes zcode.Bytes, d nano.Duration) zcode.Bytes

func AppendFloat32 ¶ 

func AppendFloat32(zb zcode.Bytes, f float32) zcode.Bytes

func AppendFloat64 ¶ 

func AppendFloat64(zb zcode.Bytes, d float64) zcode.Bytes

func AppendIP ¶ 

func AppendIP(zb zcode.Bytes, a netaddr.IP) zcode.Bytes

func AppendInt ¶ 

func AppendInt(bytes zcode.Bytes, i int64) zcode.Bytes

func AppendNet ¶ 

func AppendNet(zb zcode.Bytes, subnet *net.IPNet) zcode.Bytes

func AppendTime ¶ 

func AppendTime(bytes zcode.Bytes, t nano.Ts) zcode.Bytes

func AppendTypeValue ¶ added in v1.0.0 

func AppendTypeValue(b zcode.Bytes, t Type) zcode.Bytes

func AppendUint ¶ 

func AppendUint(bytes zcode.Bytes, i uint64) zcode.Bytes

func BuildUnion ¶ 

func BuildUnion(b *zcode.Builder, selector int, val zcode.Bytes)

BuildUnion appends to b a union described by selector and val.

func CompareTypes ¶ added in v1.0.0 

func CompareTypes(a, b Type) int

func ContainedType ¶ 

func ContainedType(typ Type) (Type, []Column)

ContainedType returns the inner type for set and array types in the first return value and the columns of its of type for record types in the second return value. ContainedType returns nil for both return values if the type is not a set, array, or record.

func DecodeBool ¶ 

func DecodeBool(zv zcode.Bytes) bool

func DecodeBytes ¶ 

func DecodeBytes(zv zcode.Bytes) []byte

func DecodeDuration ¶ 

func DecodeDuration(zv zcode.Bytes) nano.Duration

func DecodeError ¶ 

func DecodeError(zv zcode.Bytes) error

func DecodeFloat ¶ 

func DecodeFloat(zb zcode.Bytes) float64

func DecodeFloat32 ¶ 

func DecodeFloat32(zb zcode.Bytes) float32

func DecodeFloat64 ¶ 

func DecodeFloat64(zv zcode.Bytes) float64

func DecodeIP ¶ 

func DecodeIP(zv zcode.Bytes) netaddr.IP

func DecodeInt ¶ 

func DecodeInt(zv zcode.Bytes) int64

func DecodeLength ¶ added in v1.0.0 

func DecodeLength(tv zcode.Bytes) (int, zcode.Bytes)

func DecodeName ¶ added in v1.0.0 

func DecodeName(tv zcode.Bytes) (string, zcode.Bytes)

func DecodeNet ¶ 

func DecodeNet(zv zcode.Bytes) *net.IPNet

func DecodeString ¶ 

func DecodeString(zv zcode.Bytes) string

func DecodeTime ¶ 

func DecodeTime(zv zcode.Bytes) nano.Ts

func DecodeUint ¶ 

func DecodeUint(zv zcode.Bytes) uint64

func EncodeBool ¶ 

func EncodeBool(b bool) zcode.Bytes

func EncodeBytes ¶ 

func EncodeBytes(b []byte) zcode.Bytes

func EncodeDuration ¶ 

func EncodeDuration(d nano.Duration) zcode.Bytes

func EncodeError ¶ 

func EncodeError(err error) zcode.Bytes

func EncodeFloat32 ¶ 

func EncodeFloat32(d float32) zcode.Bytes

func EncodeFloat64 ¶ 

func EncodeFloat64(d float64) zcode.Bytes

func EncodeIP ¶ 

func EncodeIP(a netaddr.IP) zcode.Bytes

func EncodeInt ¶ 

func EncodeInt(i int64) zcode.Bytes

func EncodeNet ¶ 

func EncodeNet(subnet *net.IPNet) zcode.Bytes

func EncodeString ¶ 

func EncodeString(s string) zcode.Bytes

func EncodeTime ¶ 

func EncodeTime(t nano.Ts) zcode.Bytes

func EncodeTypeValue ¶ 

func EncodeTypeValue(t Type) zcode.Bytes

func EncodeUint ¶ 

func EncodeUint(i uint64) zcode.Bytes

func IDChar ¶ 

func IDChar(c rune) bool

func IsContainerType ¶ 

func IsContainerType(typ Type) bool

func IsFloat ¶ 

func IsFloat(id int) bool

True iff the type id is encoded as a float encoding. XXX add IDDecimal here when we implement coercible math with it.

func IsIdentifier ¶ 

func IsIdentifier(s string) bool

func IsInteger ¶ 

func IsInteger(id int) bool

True iff the type id is encoded as a zng signed or unsigened integer zcode.Bytes.

func IsNumber ¶ 

func IsNumber(id int) bool

True iff the type id is encoded as a zng signed or unsigned integer zcode.Bytes, float32 zcode.Bytes, or float64 zcode.Bytes.

func IsPrimitiveType ¶ 

func IsPrimitiveType(typ Type) bool

func IsRecordType ¶ 

func IsRecordType(typ Type) bool

func IsSigned ¶ 

func IsSigned(id int) bool

True iff the type id is encoded as a number encoding and is signed.

func IsTrue ¶ 

func IsTrue(zv zcode.Bytes) bool

func IsTypeName ¶ 

func IsTypeName(s string) bool

IsTypeName returns true iff s is a valid zson typedef name (exclusive of integer names for locally-scoped typedefs).

func IsUnionType ¶ 

func IsUnionType(typ Type) bool

func NormalizeMap ¶ 

func NormalizeMap(zv zcode.Bytes) zcode.Bytes

NormalizeMap interprets zv as a map body and returns an equivalent map body that is normalized according to the ZNG specification (i.e., the tag-counted value of each entry's key is lexicographically greater than that of the preceding entry).

func NormalizeSet ¶ 

func NormalizeSet(zv zcode.Bytes) zcode.Bytes

NormalizeSet interprets zv as a set body and returns an equivalent set body that is normalized according to the ZNG specification (i.e., each element's tag-counted value is lexicographically greater than that of the preceding element).

func PrimitiveName ¶ added in v1.0.0 

func PrimitiveName(typ Type) string

func QuotedName ¶ 

func QuotedName(name string) string

func QuotedString ¶ 

func QuotedString(s []byte, _ bool) string

QuotedString quotes and escapes a ZSON string for serialization in accordance with the ZSON spec. It was copied and modified [with attribution](https://github.com/brimdata/zed/blob/main/acknowledgments.txt) from the encoding/json package in the Go source code.

func TypeChar ¶ 

func TypeChar(c rune) bool

func TypeID ¶ 

func TypeID(typ Type) int

func Unhex ¶ 

func Unhex(b byte) byte

func Walk ¶ 

func Walk(typ Type, body zcode.Bytes, visit Visitor) error

Types ¶

type Allocator ¶ added in v1.0.0 

type Allocator interface {
	NewValue(Type, zcode.Bytes) *Value
	CopyValue(Value) *Value
}

type Column ¶ 

type Column struct {
	Name string
	Type Type
}

Column defines the field name and type of a column in a record type.

func NewColumn ¶ 

func NewColumn(name string, typ Type) Column

type ColumnBuilder ¶ 

type ColumnBuilder struct {
	// contains filtered or unexported fields
}

func NewColumnBuilder ¶ 

func NewColumnBuilder(zctx *Context, fields field.List) (*ColumnBuilder, error)

NewColumnBuilder constructs the zcode.Bytes representation for columns built from an array of input field selectors expressed as field.Path. Append should be called to enter field values in the left to right order of the provided fields and Encode is called to retrieve the nested zcode.Bytes value. Reset should be called before encoding the next record.

func (*ColumnBuilder) Append ¶ 

func (c *ColumnBuilder) Append(leaf []byte)

func (*ColumnBuilder) Encode ¶ 

func (c *ColumnBuilder) Encode() (zcode.Bytes, error)

func (*ColumnBuilder) Reset ¶ 

func (c *ColumnBuilder) Reset()

func (*ColumnBuilder) TypedColumns ¶ 

func (c *ColumnBuilder) TypedColumns(types []Type) []Column

A ColumnBuilder understands the shape of a sequence of FieldExprs (i.e., which columns are inside nested records) but not the types. TypedColumns takes an array of Types for the individual fields and constructs an array of Columns that reflects the fullly typed structure. This is suitable for e.g. allocating a descriptor.

type Context ¶ 

type Context struct {
	// contains filtered or unexported fields
}

A Context implements the "type context" in the Zed model. For a given set of related Values, each Value has a type from a shared Context. The Context manages the transitive closure of Types so that each unique type corresponds to exactly one Type pointer allowing type equivlance to be determined by pointer comparison. (Type pointers from distinct Contexts obviously do not have this property.) A Context also provides an efficient means to translate type values (represented as serialized ZNG) to Types. This provides an efficient means to translate Type pointers from one context to another.

func NewContext ¶ 

func NewContext() *Context

func (*Context) AddColumns ¶ 

func (c *Context) AddColumns(r *Value, newCols []Column, vals []Value) (*Value, error)

AddColumns returns a new Record with columns equal to the given record along with new rightmost columns as indicated with the given values. If any of the newly provided columns already exists in the specified value, an error is returned.

func (*Context) DecodeTypeValue ¶ added in v1.0.0 

func (c *Context) DecodeTypeValue(tv zcode.Bytes) (Type, zcode.Bytes)

func (*Context) Lookup ¶ 

func (c *Context) Lookup(id int) *TypeRecord

func (*Context) LookupByValue ¶ 

func (c *Context) LookupByValue(tv zcode.Bytes) (Type, error)

LookupByValue returns the Type indicated by a binary-serialized type value. This provides a means to translate a type-context-independent serialized encoding for an arbitrary type into the reciever Context.

func (*Context) LookupType ¶ 

func (c *Context) LookupType(id int) (Type, error)

func (*Context) LookupTypeArray ¶ 

func (c *Context) LookupTypeArray(inner Type) *TypeArray

func (*Context) LookupTypeDef ¶ 

func (c *Context) LookupTypeDef(name string) *TypeNamed

func (*Context) LookupTypeEnum ¶ 

func (c *Context) LookupTypeEnum(symbols []string) *TypeEnum

func (*Context) LookupTypeError ¶ added in v1.0.0 

func (c *Context) LookupTypeError(inner Type) *TypeError

func (*Context) LookupTypeMap ¶ 

func (c *Context) LookupTypeMap(keyType, valType Type) *TypeMap

func (*Context) LookupTypeNamed ¶ added in v1.0.0 

func (c *Context) LookupTypeNamed(name string, target Type) (*TypeNamed, error)

func (*Context) LookupTypeRecord ¶ 

func (c *Context) LookupTypeRecord(columns []Column) (*TypeRecord, error)

LookupTypeRecord returns a TypeRecord within this context that binds with the indicated columns. Subsequent calls with the same columns will return the same record pointer. If the type doesn't exist, it's created, stored, and returned. The closure of types within the columns must all be from this type context. If you want to use columns from a different type context, use TranslateTypeRecord.

func (*Context) LookupTypeSet ¶ 

func (c *Context) LookupTypeSet(inner Type) *TypeSet

func (*Context) LookupTypeUnion ¶ 

func (c *Context) LookupTypeUnion(types []Type) *TypeUnion

func (*Context) LookupTypeValue ¶ 

func (c *Context) LookupTypeValue(typ Type) *Value

func (*Context) Missing ¶ added in v1.0.0 

func (c *Context) Missing() *Value

func (*Context) MustLookupTypeRecord ¶ 

func (c *Context) MustLookupTypeRecord(columns []Column) *TypeRecord

func (*Context) NewError ¶ added in v1.0.0 

func (c *Context) NewError(err error) *Value

func (*Context) NewErrorf ¶ added in v1.0.0 

func (c *Context) NewErrorf(format string, args ...interface{}) *Value

func (*Context) Quiet ¶ added in v1.0.0 

func (c *Context) Quiet() *Value

func (*Context) Reset ¶ 

func (c *Context) Reset()

func (*Context) StringTypeError ¶ added in v1.0.0 

func (c *Context) StringTypeError() *TypeError

func (*Context) TranslateType ¶ 

func (c *Context) TranslateType(ext Type) (Type, error)

TranslateType takes a type from another context and creates and returns that type in this context.

func (*Context) TranslateTypeRecord ¶ 

func (t *Context) TranslateTypeRecord(ext *TypeRecord) (*TypeRecord, error)

func (*Context) WrapError ¶ added in v1.0.0 

func (c *Context) WrapError(msg string, val *Value) *Value

type Kind ¶ added in v1.0.0 

type Kind int
const (
	PrimitiveKind Kind = iota
	RecordKind
	ArrayKind
	SetKind
	MapKind
	UnionKind
	EnumKind
	ErrorKind
)

func (Kind) String ¶ added in v1.0.0 

func (k Kind) String() string

type Mapper ¶ 

type Mapper struct {
	// contains filtered or unexported fields
}

func NewMapper ¶ 

func NewMapper(out *Context) *Mapper

func (*Mapper) Enter ¶ 

func (m *Mapper) Enter(id int, ext Type) (Type, error)

func (*Mapper) EnterType ¶ 

func (m *Mapper) EnterType(id int, typ Type)

func (*Mapper) Lookup ¶ 

func (m *Mapper) Lookup(id int) Type

Lookup tranlates Zed types by type ID from one context to another. The first context is implied by the argument to Lookup() and the output type context is explicitly determined by the argument to NewMapper(). If a binding has not yet been entered, nil is returned and Enter() should be called to create the binding. There is a race here when two threads attempt to update the same ID, but it is safe because the outputContext will return the same the pointer so the second update does not change anything.

type Type ¶ 

type Type interface {
	// ID returns a unique (per Context) identifier that
	// represents this type.  For a named type, this identifier
	// represents the underlying type and not the named type itself.
	// Callers that care about the underlying type of a Value for
	// example should prefer to use this instead of using a Go
	// type assertion on a Type instance.
	ID() int
	Kind() Kind
}

A Type is an interface presented by a zeek type. Types can be used to infer type compatibility and create new values of the underlying type.

func InnerType ¶ 

func InnerType(typ Type) Type

InnerType returns the element type for the underlying set or array type or nil if the underlying type is not a set or array.

func LookupPrimitive ¶ 

func LookupPrimitive(name string) Type

func LookupPrimitiveByID ¶ 

func LookupPrimitiveByID(id int) Type

func TypeUnder ¶ added in v1.0.0 

func TypeUnder(typ Type) Type

type TypeArray ¶ 

type TypeArray struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeArray ¶ 

func NewTypeArray(id int, typ Type) *TypeArray

func (*TypeArray) ID ¶ 

func (t *TypeArray) ID() int

func (*TypeArray) Kind ¶ added in v1.0.0 

func (t *TypeArray) Kind() Kind

type TypeEnum ¶ 

type TypeEnum struct {
	Symbols []string
	// contains filtered or unexported fields
}

func NewTypeEnum ¶ 

func NewTypeEnum(id int, symbols []string) *TypeEnum

func (*TypeEnum) ID ¶ 

func (t *TypeEnum) ID() int

func (*TypeEnum) Kind ¶ added in v1.0.0 

func (t *TypeEnum) Kind() Kind

func (*TypeEnum) Lookup ¶ 

func (t *TypeEnum) Lookup(symbol string) int

func (*TypeEnum) Symbol ¶ 

func (t *TypeEnum) Symbol(index int) (string, error)

type TypeError ¶ 

type TypeError struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeError ¶ added in v1.0.0 

func NewTypeError(id int, typ Type) *TypeError

func (*TypeError) ID ¶ added in v1.0.0 

func (t *TypeError) ID() int

func (*TypeError) IsMissing ¶ added in v1.0.0 

func (t *TypeError) IsMissing(zv zcode.Bytes) bool

func (*TypeError) IsQuiet ¶ added in v1.0.0 

func (t *TypeError) IsQuiet(zv zcode.Bytes) bool

func (*TypeError) Kind ¶ added in v1.0.0 

func (t *TypeError) Kind() Kind

type TypeMap ¶ 

type TypeMap struct {
	KeyType Type
	ValType Type
	// contains filtered or unexported fields
}

func NewTypeMap ¶ 

func NewTypeMap(id int, keyType, valType Type) *TypeMap

func (*TypeMap) Decode ¶ 

func (t *TypeMap) Decode(zv zcode.Bytes) (Value, Value, error)

func (*TypeMap) ID ¶ 

func (t *TypeMap) ID() int

func (*TypeMap) Kind ¶ added in v1.0.0 

func (t *TypeMap) Kind() Kind

type TypeNamed ¶ added in v1.0.0 

type TypeNamed struct {
	Name string
	Type Type
	// contains filtered or unexported fields
}

func NewTypeNamed ¶ added in v1.0.0 

func NewTypeNamed(id int, name string, typ Type) *TypeNamed

func (*TypeNamed) ID ¶ added in v1.0.0 

func (t *TypeNamed) ID() int

func (*TypeNamed) Kind ¶ added in v1.0.0 

func (t *TypeNamed) Kind() Kind

func (*TypeNamed) NamedID ¶ added in v1.0.0 

func (t *TypeNamed) NamedID() int

type TypeOfBool ¶ 

type TypeOfBool struct{}

func (*TypeOfBool) ID ¶ 

func (t *TypeOfBool) ID() int

func (*TypeOfBool) Kind ¶ added in v1.0.0 

func (t *TypeOfBool) Kind() Kind

type TypeOfBytes ¶ 

type TypeOfBytes struct{}

func (*TypeOfBytes) Format ¶ 

func (t *TypeOfBytes) Format(zv zcode.Bytes) string

func (*TypeOfBytes) ID ¶ 

func (t *TypeOfBytes) ID() int

func (*TypeOfBytes) Kind ¶ added in v1.0.0 

func (t *TypeOfBytes) Kind() Kind

type TypeOfDuration ¶ 

type TypeOfDuration struct{}

func (*TypeOfDuration) ID ¶ 

func (t *TypeOfDuration) ID() int

func (*TypeOfDuration) Kind ¶ added in v1.0.0 

func (t *TypeOfDuration) Kind() Kind

type TypeOfFloat32 ¶ 

type TypeOfFloat32 struct{}

func (*TypeOfFloat32) ID ¶ 

func (t *TypeOfFloat32) ID() int

func (*TypeOfFloat32) Kind ¶ added in v1.0.0 

func (t *TypeOfFloat32) Kind() Kind

func (*TypeOfFloat32) Marshal ¶ 

func (t *TypeOfFloat32) Marshal(zb zcode.Bytes) interface{}

type TypeOfFloat64 ¶ 

type TypeOfFloat64 struct{}

func (*TypeOfFloat64) ID ¶ 

func (t *TypeOfFloat64) ID() int

func (*TypeOfFloat64) Kind ¶ added in v1.0.0 

func (t *TypeOfFloat64) Kind() Kind

func (*TypeOfFloat64) Marshal ¶ 

func (t *TypeOfFloat64) Marshal(zv zcode.Bytes) interface{}

type TypeOfIP ¶ 

type TypeOfIP struct{}

func (*TypeOfIP) ID ¶ 

func (t *TypeOfIP) ID() int

func (*TypeOfIP) Kind ¶ added in v1.0.0 

func (t *TypeOfIP) Kind() Kind

type TypeOfInt16 ¶ 

type TypeOfInt16 struct{}

func (*TypeOfInt16) ID ¶ 

func (t *TypeOfInt16) ID() int

func (*TypeOfInt16) Kind ¶ added in v1.0.0 

func (t *TypeOfInt16) Kind() Kind

type TypeOfInt32 ¶ 

type TypeOfInt32 struct{}

func (*TypeOfInt32) ID ¶ 

func (t *TypeOfInt32) ID() int

func (*TypeOfInt32) Kind ¶ added in v1.0.0 

func (t *TypeOfInt32) Kind() Kind

type TypeOfInt64 ¶ 

type TypeOfInt64 struct{}

func (*TypeOfInt64) ID ¶ 

func (t *TypeOfInt64) ID() int

func (*TypeOfInt64) Kind ¶ added in v1.0.0 

func (t *TypeOfInt64) Kind() Kind

type TypeOfInt8 ¶ 

type TypeOfInt8 struct{}

func (*TypeOfInt8) ID ¶ 

func (t *TypeOfInt8) ID() int

func (*TypeOfInt8) Kind ¶ added in v1.0.0 

func (t *TypeOfInt8) Kind() Kind

type TypeOfNet ¶ 

type TypeOfNet struct{}

func (*TypeOfNet) ID ¶ 

func (t *TypeOfNet) ID() int

func (*TypeOfNet) Kind ¶ added in v1.0.0 

func (t *TypeOfNet) Kind() Kind

type TypeOfNull ¶ 

type TypeOfNull struct{}

func (*TypeOfNull) ID ¶ 

func (t *TypeOfNull) ID() int

func (*TypeOfNull) Kind ¶ added in v1.0.0 

func (t *TypeOfNull) Kind() Kind

type TypeOfString ¶ 

type TypeOfString struct{}

func (*TypeOfString) ID ¶ 

func (t *TypeOfString) ID() int

func (*TypeOfString) Kind ¶ added in v1.0.0 

func (t *TypeOfString) Kind() Kind

type TypeOfTime ¶ 

type TypeOfTime struct{}

func (*TypeOfTime) ID ¶ 

func (t *TypeOfTime) ID() int

func (*TypeOfTime) Kind ¶ added in v1.0.0 

func (t *TypeOfTime) Kind() Kind

type TypeOfType ¶ 

type TypeOfType struct{}

func (*TypeOfType) ID ¶ 

func (t *TypeOfType) ID() int

func (*TypeOfType) Kind ¶ added in v1.0.0 

func (t *TypeOfType) Kind() Kind

type TypeOfUint16 ¶ 

type TypeOfUint16 struct{}

func (*TypeOfUint16) ID ¶ 

func (t *TypeOfUint16) ID() int

func (*TypeOfUint16) Kind ¶ added in v1.0.0 

func (t *TypeOfUint16) Kind() Kind

type TypeOfUint32 ¶ 

type TypeOfUint32 struct{}

func (*TypeOfUint32) ID ¶ 

func (t *TypeOfUint32) ID() int

func (*TypeOfUint32) Kind ¶ added in v1.0.0 

func (t *TypeOfUint32) Kind() Kind

type TypeOfUint64 ¶ 

type TypeOfUint64 struct{}

func (*TypeOfUint64) ID ¶ 

func (t *TypeOfUint64) ID() int

func (*TypeOfUint64) Kind ¶ added in v1.0.0 

func (t *TypeOfUint64) Kind() Kind

type TypeOfUint8 ¶ 

type TypeOfUint8 struct{}

func (*TypeOfUint8) ID ¶ 

func (t *TypeOfUint8) ID() int

func (*TypeOfUint8) Kind ¶ added in v1.0.0 

func (t *TypeOfUint8) Kind() Kind

type TypeRecord ¶ 

type TypeRecord struct {
	Columns []Column
	LUT     map[string]int
	// contains filtered or unexported fields
}

func NewTypeRecord ¶ 

func NewTypeRecord(id int, columns []Column) *TypeRecord

func TypeRecordOf ¶ 

func TypeRecordOf(typ Type) *TypeRecord

func (*TypeRecord) ColumnOfField ¶ 

func (t *TypeRecord) ColumnOfField(field string) (int, bool)

func (*TypeRecord) Decode ¶ 

func (t *TypeRecord) Decode(zv zcode.Bytes) ([]Value, error)

XXX we shouldn't need this... tests are using it

func (*TypeRecord) HasField ¶ 

func (t *TypeRecord) HasField(field string) bool

func (*TypeRecord) ID ¶ 

func (t *TypeRecord) ID() int

func (*TypeRecord) Kind ¶ added in v1.0.0 

func (t *TypeRecord) Kind() Kind

func (*TypeRecord) Marshal ¶ 

func (t *TypeRecord) Marshal(zv zcode.Bytes) interface{}

func (*TypeRecord) TypeOfField ¶ 

func (t *TypeRecord) TypeOfField(field string) (Type, bool)

type TypeSet ¶ 

type TypeSet struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeSet ¶ 

func NewTypeSet(id int, typ Type) *TypeSet

func (*TypeSet) ID ¶ 

func (t *TypeSet) ID() int

func (*TypeSet) Kind ¶ added in v1.0.0 

func (t *TypeSet) Kind() Kind

type TypeUnion ¶ 

type TypeUnion struct {
	Types []Type
	LUT   map[Type]int
	// contains filtered or unexported fields
}

func NewTypeUnion ¶ 

func NewTypeUnion(id int, types []Type) *TypeUnion

func (*TypeUnion) ID ¶ 

func (t *TypeUnion) ID() int

func (*TypeUnion) Kind ¶ added in v1.0.0 

func (t *TypeUnion) Kind() Kind

func (*TypeUnion) Selector ¶ added in v1.0.0 

func (t *TypeUnion) Selector(typ Type) int

Selector returns the selector for typ in the union. If no type exists -1 is returned.

func (*TypeUnion) SplitZNG ¶ added in v1.0.0 

func (t *TypeUnion) SplitZNG(zv zcode.Bytes) (Type, int64, zcode.Bytes, error)

SplitZNG takes a zng encoding of a value of the receiver's union type and returns the concrete type of the value, its selector, and the value encoding.

func (*TypeUnion) Type ¶ 

func (t *TypeUnion) Type(selector int) (Type, error)

Type returns the type corresponding to selector.

type TypeVectorTable ¶ 

type TypeVectorTable struct {
	// contains filtered or unexported fields
}

func NewTypeVectorTable ¶ 

func NewTypeVectorTable() *TypeVectorTable

func (*TypeVectorTable) Length ¶ 

func (t *TypeVectorTable) Length() int

func (*TypeVectorTable) Lookup ¶ 

func (t *TypeVectorTable) Lookup(types []Type) int

func (*TypeVectorTable) LookupByValues ¶ 

func (t *TypeVectorTable) LookupByValues(vals []Value) int

func (*TypeVectorTable) Types ¶ 

func (t *TypeVectorTable) Types(id int) []Type

type Value ¶ 

type Value struct {
	Type  Type
	Bytes zcode.Bytes
}

func NewBool ¶ 

func NewBool(b bool) *Value

func NewBytes ¶ 

func NewBytes(b []byte) *Value

func NewDuration ¶ 

func NewDuration(d nano.Duration) *Value

func NewFloat32 ¶ 

func NewFloat32(f float32) *Value

func NewFloat64 ¶ 

func NewFloat64(f float64) *Value

func NewIP ¶ 

func NewIP(a netaddr.IP) *Value

func NewNet ¶ 

func NewNet(s *net.IPNet) *Value

func NewString ¶ 

func NewString(s string) *Value

func NewTime ¶ 

func NewTime(ts nano.Ts) *Value

func NewTypeValue ¶ 

func NewTypeValue(t Type) *Value

func NewUint64 ¶ 

func NewUint64(v uint64) *Value

func NewValue ¶ added in v0.32.0 

func NewValue(zt Type, zb zcode.Bytes) *Value

func Not ¶ added in v0.32.0 

func Not(zb zcode.Bytes) *Value

Not returns the inverse Value of the Boolean-typed bytes value of zb.

func (*Value) ArrayIndex ¶ 

func (v *Value) ArrayIndex(idx int64) (Value, error)

If the passed-in element is an array, attempt to get the idx'th element, and return its type and raw representation. Returns an error if the passed-in element is not an array or if idx is outside the array bounds.

func (*Value) AsBool ¶ added in v1.0.0 

func (v *Value) AsBool() bool

func (*Value) AsInt ¶ added in v1.0.0 

func (v *Value) AsInt() int64

func (*Value) AsString ¶ added in v1.0.0 

func (v *Value) AsString() string

func (*Value) AsTime ¶ added in v1.0.0 

func (v *Value) AsTime() nano.Ts

func (*Value) ColumnOfField ¶ added in v0.32.0 

func (v *Value) ColumnOfField(field string) (int, bool)

func (*Value) Columns ¶ added in v0.32.0 

func (r *Value) Columns() []Column

func (*Value) ContainerLength ¶ 

func (v *Value) ContainerLength() (int, error)

func (*Value) Copy ¶ 

func (v *Value) Copy() *Value

Copy returns a copy of v that does not share v.Bytes. The copy's Bytes field is nil if and only if v.Bytes is nil.

func (*Value) CopyFrom ¶ added in v1.0.0 

func (v *Value) CopyFrom(from *Value)

CopyFrom copies from into v, reusing v.Bytes if possible and setting v.Bytes to nil if and only if from.Bytes is nil.

func (*Value) Deref ¶ added in v0.32.0 

func (v *Value) Deref(field string) *Value

func (*Value) DerefByColumn ¶ added in v1.0.0 

func (v *Value) DerefByColumn(col int) *Value

func (*Value) DerefPath ¶ added in v1.0.0 

func (v *Value) DerefPath(path field.Path) *Value

func (*Value) Elements ¶ 

func (v *Value) Elements() ([]Value, error)

Elements returns an array of Values for the given container type. Returns an error if the element is not an array or set.

func (*Value) Encode ¶ 

func (v *Value) Encode(dst zcode.Bytes) zcode.Bytes

Encode appends the ZNG representation of this value to the passed in argument and returns the resulting zcode.Bytes (which may or may not be the same underlying buffer, as with append(), depending on its capacity)

func (*Value) Equal ¶ 

func (v *Value) Equal(p Value) bool

func (*Value) HasField ¶ added in v0.32.0 

func (r *Value) HasField(field string) bool

func (*Value) IsContainer ¶ 

func (v *Value) IsContainer() bool

func (*Value) IsError ¶ 

func (v *Value) IsError() bool

func (*Value) IsMissing ¶ 

func (v *Value) IsMissing() bool

func (*Value) IsNull ¶ added in v1.0.0 

func (v *Value) IsNull() bool

IsNull returns true if and only if v is a null value of any type.

func (*Value) IsQuiet ¶ added in v1.0.0 

func (v *Value) IsQuiet() bool

func (*Value) IsString ¶ added in v1.0.0 

func (v *Value) IsString() bool

func (*Value) Iter ¶ 

func (v *Value) Iter() zcode.Iter

func (*Value) MissingAsNull ¶ added in v1.0.0 

func (v *Value) MissingAsNull() *Value

func (*Value) String ¶ 

func (v *Value) String() string

String implements fmt.Stringer.String. It should only be used for logs, debugging, etc. Any caller that requires a specific output format should use FormatAs() instead.

func (*Value) Walk ¶ added in v0.32.0 

func (r *Value) Walk(rv Visitor) error

Walk traverses a value in depth-first order, calling a Visitor on the way.

type Visitor ¶ 

type Visitor func(typ Type, body zcode.Bytes) error

A Visitor is called for each value in a record encountered by Walk. If the visitor returns an error, the walk stops and that error will be returned to the caller of Walk(). The sole exception is when the visitor returns the special value SkipContainer.

Source Files ¶

View all Source files

Directories ¶

Path Synopsis
api
client
client/auth0
queryio
cli
auto
cacheflags
inputflags
lakeflags
outputflags
procflags
queryflags
zq
cmd
gentoken
mockbrim
zc
zed
zed/auth
zed/branch
zed/create
zed/delete
zed/dev
zed/dev/compile
zed/dev/dig
zed/dev/dig/frames
zed/dev/dig/section
zed/dev/dig/slice
zed/dev/dig/trailer
zed/dev/indexfile
zed/dev/indexfile/create
zed/dev/indexfile/lookup
zed/dev/zst
zed/dev/zst/create
zed/dev/zst/cut
zed/dev/zst/read
zed/drop
zed/index
zed/init
zed/load
zed/log
zed/ls
zed/merge
zed/query
zed/rename
zed/revert
zed/root
zed/serve
zed/use
zed/vacate
zq
zst
ast
Package ast declares the types used to represent syntax trees for Zed queries.
 Package ast declares the types used to represent syntax trees for Zed queries.
ast/dag
ast/zed
kernel
optimizer
parser
semantic
Package index provides an API for creating, merging, indexing, and querying Zed indexes.
 Package index provides an API for creating, merging, indexing, and querying Zed indexes.
api
branches
commits
data
index
journal
mock
pools
seekindex
Package mdtest finds example shell commands in Markdown files and runs them, checking for expected output and exit status.
 Package mdtest finds example shell commands in Markdown files and runs them, checking for expected output and exit status.
pkg
anymath
bufwriter
Package bufwriter provides a wrapper for a io.WriteCloser that uses buffered output via a bufio.Writer and calls Flush on close.
 Package bufwriter provides a wrapper for a io.WriteCloser that uses buffered output via a bufio.Writer and calls Flush on close.
byteconv
Package byteconv implements conversions from byte slice representations of various data types.
 Package byteconv implements conversions from byte slice representations of various data types.
charm
Package charm is minimilast CLI framework inspired by cobra and urfave/cli.
 Package charm is minimilast CLI framework inspired by cobra and urfave/cli.
colw
Package colw lays out columns for display of a list when you don't know ahead of time how many columns should exist.
 Package colw lays out columns for display of a list when you don't know ahead of time how many columns should exist.
display
field
fs
glob
Package glob implements glob-style pattern matching
 Package glob implements glob-style pattern matching
httpd
jsonlexer
nano
peeker
promtest
repl
Package repl is a simple read-eval-print loop.
 Package repl is a simple read-eval-print loop.
rlimit
Package rlimit provides a single function, RaiseOpenFilesLimit.
 Package rlimit provides a single function, RaiseOpenFilesLimit.
s3io
skim
storage
storage/cache
Package cache contains facilities for caching immutable files, typically for a cloud object store.
 Package cache contains facilities for caching immutable files, typically for a cloud object store.
storage/mock
Package mock is a generated GoMock package.
 Package mock is a generated GoMock package.
stringsearch
terminal
terminal/color
test
units
unpack
expr
expr/agg
expr/coerce
expr/extent
expr/function
expr/result
op
op/combine
A combine proc merges multiple upstream inputs into one output.
 A combine proc merges multiple upstream inputs into one output.
op/explode
op/exprswitch
op/fork
op/from
op/fuse
op/groupby
op/head
op/join
op/merge
op/pass
op/shape
op/sort
op/spill
op/switcher
op/tail
op/top
op/traverse
op/uniq
op/yield
auth
logger
Package zcode implements serialization and deserialzation for ZNG values.
 Package zcode implements serialization and deserialzation for ZNG values.
zio
anyio
csvio
emitter
jsonio
lakeio
parquetio
tableio
textio
zeekio
zjsonio
zng21io
Package zng21io provides low performance, read-only for the old ZNG format prior to the changes introduced in January 2021.
 Package zng21io provides low performance, read-only for the old ZNG format prior to the changes introduced in January 2021.
zng21io/zed21
zngio
Package zngio provides an API for reading and writing zng values and directives in binary zng format.
 Package zngio provides an API for reading and writing zng values and directives in binary zng format.
zsonio
zstio
Package zqe provides a mechanism to create or wrap errors with information that will aid in reporting them to users and returning them to api callers.
 Package zqe provides a mechanism to create or wrap errors with information that will aid in reporting them to users and returning them to api callers.
Package zson provides fundamental interfaces to the ZSON data format comprising Reader, Writer, Parser, and so forth.
 Package zson provides fundamental interfaces to the ZSON data format comprising Reader, Writer, Parser, and so forth.
zst
Package zst implements the reading and writing of ZST storage objects to and from any Zed format.
 Package zst implements the reading and writing of ZST storage objects to and from any Zed format.
column
Package column implements the organization of columns on storage for a ZST columnar storage object.
 Package column implements the organization of columns on storage for a ZST columnar storage object.
Package ztest runs formulaic tests ("ztests") that can be (1) run in-process with the compiled-ini zq code base, (2) run as a sub-process using the zq executable build artifact, or (3) run as a bash script running a sequence of arbitrary shell commands invoking any of the build artifacts.
 Package ztest runs formulaic tests ("ztests") that can be (1) run in-process with the compiled-ini zq code base, (2) run as a sub-process using the zq executable build artifact, or (3) run as a bash script running a sequence of arbitrary shell commands invoking any of the build artifacts.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.