Documentation
¶
Overview ¶
Package coder contains coder representation and utilities. Coders describe how to serialize and deserialize pipeline data and may be provided by users.
Index ¶
- Variables
- func DecodeEventTime(r io.Reader) (typex.EventTime, error)
- func DecodeInt32(r io.Reader) (int32, error)
- func DecodeUint32(r io.Reader) (uint32, error)
- func DecodeUint64(r io.Reader) (uint64, error)
- func DecodeVarInt(r io.Reader) (int32, error)
- func DecodeVarUint64(r io.Reader) (uint64, error)
- func EncodeEventTime(et typex.EventTime, w io.Writer) error
- func EncodeInt32(value int32, w io.Writer) error
- func EncodeUint32(value uint32, w io.Writer) error
- func EncodeUint64(value uint64, w io.Writer) error
- func EncodeVarInt(value int32, w io.Writer) error
- func EncodeVarUint64(value uint64, w io.Writer) error
- func IsCoGBK(c *Coder) bool
- func IsKV(c *Coder) bool
- func IsW(c *Coder) bool
- func Types(list []*Coder) []typex.FullType
- type Coder
- type CustomCoder
- type Kind
- type WindowCoder
- type WindowKind
Constants ¶
This section is empty.
Variables ¶
var ErrVarIntTooLong = errors.New("varint too long")
ErrVarIntTooLong indicates a data corruption issue that needs special handling by callers of decode. TODO(herohde): have callers perform this special handling.
Functions ¶
func DecodeEventTime ¶
DecodeEventTime decodes an EventTime.
func DecodeInt32 ¶
DecodeInt32 decodes an int32 in big endian format.
func DecodeUint32 ¶
DecodeUint32 decodes an uint32 in big endian format.
func DecodeUint64 ¶
DecodeUint64 decodes an uint64 in big endian format.
func DecodeVarUint64 ¶
DecodeVarUint64 decodes an uint64.
func EncodeEventTime ¶
EncodeEventTime encodes an EventTime as an uint64. The encoding is millis-since-epoch, but shifted so that the byte representation of negative values are lexicographically ordered before the byte representation of positive values.
func EncodeInt32 ¶
EncodeInt32 encodes an int32 in big endian format.
func EncodeUint32 ¶
EncodeUint32 encodes an uint32 in big endian format.
func EncodeUint64 ¶
EncodeUint64 encodes an uint64 in big endian format.
func EncodeVarInt ¶
EncodeVarInt encodes an int32.
func EncodeVarUint64 ¶
EncodeVarUint64 encodes an uint64.
Types ¶
type Coder ¶
type Coder struct {
Kind Kind
T typex.FullType
Components []*Coder // WindowedValue, KV, CoGBK
Custom *CustomCoder // Custom
Window *WindowCoder // WindowedValue
}
Coder is a description of how to encode and decode values of a given type. Except for the "custom" kind, they are built in and must adhere to the (unwritten) Beam specification.
func NewBytes ¶
func NewBytes() *Coder
NewBytes returns a new []byte coder using the built-in scheme. It is always nested, for now.
func NewVarInt ¶
func NewVarInt() *Coder
NewVarInt returns a new int32 coder using the built-in scheme.
func NewW ¶
func NewW(c *Coder, w *WindowCoder) *Coder
NewW returns a WindowedValue coder for the window of elements.
func SkipW ¶
SkipW returns the data coder used by a WindowedValue, or returns the coder. This allows code to seamlessly traverse WindowedValues without additional conditional code.
type CustomCoder ¶
type CustomCoder struct {
// Name is the coder name. Informational only.
Name string
// Type is the underlying concrete type that is being coded. It is
// available to Enc and Dec. It must be a concrete type.
Type reflect.Type
// Enc is the encoding function : T -> []byte. It may optionally take a
// reflect.Type parameter and return an error as well.
Enc *funcx.Fn
// Dec is the decoding function: []byte -> T. It may optionally take a
// reflect.Type parameter and return an error as well.
Dec *funcx.Fn
}
CustomCoder contains possibly untyped encode/decode user functions that are type-bound at runtime. Universal coders can thus be used for many different types, but each CustomCoder instance will be bound to a specific type.
func NewCustomCoder ¶
func NewCustomCoder(id string, t reflect.Type, encode, decode interface{}) (*CustomCoder, error)
NewCustomCoder creates a coder for the supplied parameters defining a particular encoding strategy.
func (*CustomCoder) Equals ¶
func (c *CustomCoder) Equals(o *CustomCoder) bool
Equals returns true iff the two custom coders are equal. It assumes that functions with the same name and types are identical.
func (*CustomCoder) String ¶
func (c *CustomCoder) String() string
type Kind ¶
type Kind string
Kind represents the type of coder used.
const ( Custom Kind = "Custom" // Implicitly length-prefixed Bytes Kind = "bytes" // Implicitly length-prefixed as part of the encoding VarInt Kind = "varint" WindowedValue Kind = "W" KV Kind = "KV" // CoGBK is currently equivalent to either // // KV<X,Iterable<Y>> (if GBK) // KV<X,Iterable<KV<int,Y>>> (if CoGBK, using a tagged union encoding) // // It requires special handling in translation to the model pipeline in the latter case // to add the incoming index for each input. // // TODO(BEAM-490): once this JIRA is done, this coder should become the new thing. CoGBK Kind = "CoGBK" )
Tags for the various Beam encoding strategies. https://beam.apache.org/documentation/programming-guide/#coders documents the usage of coders in the Beam environment.
type WindowCoder ¶
type WindowCoder struct {
Kind WindowKind
}
WindowCoder represents a Window coder.
func NewGlobalWindow ¶
func NewGlobalWindow() *WindowCoder
NewGlobalWindow returns a window coder for the global window.
func NewIntervalWindow ¶
func NewIntervalWindow() *WindowCoder
NewIntervalWindow returns a window coder for interval windows.
func (*WindowCoder) Equals ¶
func (w *WindowCoder) Equals(o *WindowCoder) bool
func (*WindowCoder) String ¶
func (w *WindowCoder) String() string
type WindowKind ¶
type WindowKind string
WindowKind represents a kind of window coder.
const ( GlobalWindow WindowKind = "GWC" IntervalWindow WindowKind = "IWC" )
Source Files