README
¶
avro
avro is a decoder for Apache AVRO that decodes directly into Go structs and follows naming from JSON tags. It is intended primarily for decoding output from Google's Big Query.
Documentation
¶
Overview ¶
Package avro is an AVRO decoder aimed principly at decoding AVRO output from Google's Big Query. It decodes directly into Go structs, and uses json tags as naming hints.
The primary interface to the package is ReadFile. This reads an AVRO file, combining the schema in the file with type information from the struct passed via the out parameter to decode the records. It then passes an instance of a struct of type out to the callback cb for each record in the file.
You can implement custom decoders for your own types and register them via the Register function. github.com/phil/avro/null is an example of custom decoders for the types defined in github.com/unravelin/null
Index ¶
- func ReadFile(r Reader, out interface{}, cb func(val unsafe.Pointer) error) error
- func Register(typ reflect.Type, f CodecBuildFunc)
- type BoolCodec
- type BytesCodec
- type Codec
- type CodecBuildFunc
- type DoubleCodec
- type FileHeader
- type Float32DoubleCodec
- type FloatCodec
- type Int16Codec
- type Int32Codec
- type Int64Codec
- type MapCodec
- type Reader
- type Schema
- type SchemaObject
- type SchemaRecordField
- type StringCodec
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ReadFile ¶
ReadFile reads from an AVRO file. The records in the file are decoded into structs of the type indicated by out. These are fed back to the application via the cb callback. ReadFile calls cb with a pointer to the struct. The pointer is converted to an unsafe.Pointer. The pointer should not be retained by the application past the return of cb.
var records []myrecord
if err := ReadFile(f, myrecord{}, func(val unsafe.Pointer) error {
records = append(records, *(*record)(val))
return nil
}); err != nil {
return err
}
func Register ¶
func Register(typ reflect.Type, f CodecBuildFunc)
Register is used to set a custom codec builder for a type
Types ¶
type BytesCodec ¶
type BytesCodec struct{}
func (BytesCodec) New ¶
func (BytesCodec) New() unsafe.Pointer
func (BytesCodec) Skip ¶
func (BytesCodec) Skip(r Reader) error
type Codec ¶
type Codec interface {
// Read reads the wire format bytes for the current field from r and sets up
// the value that p points to. The codec can assume that the memory for an
// instance of the type for which the codec is registered is present behind
// p
Read(r Reader, p unsafe.Pointer) error
// Skip advances the reader over the bytes for the current field.
Skip(r Reader) error
// New creates a pointer to the type for which the codec is registered. It is
// used if the enclosing record has a field that is a pointer to this type
New() unsafe.Pointer
}
Codec defines a decoder for a type. It may eventually define an encoder too. You can write custom Codecs for types. See Register and CodecBuildFunc
type CodecBuildFunc ¶
CodecBuildFunc is the function signature for a codec builder. If you want to customise AVRO decoding for a type register a CodecBuildFunc via the Register call. Schema is the AVRO schema for the type to build. typ should match the type the function was registered under.
type DoubleCodec ¶
type DoubleCodec struct{}
func (DoubleCodec) New ¶
func (DoubleCodec) New() unsafe.Pointer
func (DoubleCodec) Skip ¶
func (DoubleCodec) Skip(r Reader) error
type FileHeader ¶
type FileHeader struct {
Magic [4]byte `json:"magic"`
Meta map[string][]byte `json:"meta"`
Sync [16]byte `json:"sync"`
}
FileHeader represents an AVRO file header
type Float32DoubleCodec ¶
type Float32DoubleCodec struct {
DoubleCodec
}
func (Float32DoubleCodec) New ¶
func (Float32DoubleCodec) New() unsafe.Pointer
type FloatCodec ¶
type FloatCodec struct{}
func (FloatCodec) New ¶
func (FloatCodec) New() unsafe.Pointer
func (FloatCodec) Skip ¶
func (FloatCodec) Skip(r Reader) error
type Int16Codec ¶
type Int16Codec struct{}
func (Int16Codec) New ¶
func (Int16Codec) New() unsafe.Pointer
func (Int16Codec) Skip ¶
func (Int16Codec) Skip(r Reader) error
type Int32Codec ¶
type Int32Codec struct{}
func (Int32Codec) New ¶
func (Int32Codec) New() unsafe.Pointer
func (Int32Codec) Skip ¶
func (Int32Codec) Skip(r Reader) error
type MapCodec ¶
type MapCodec struct {
// contains filtered or unexported fields
}
MapCodec is a decoder for map types. The key must always be string
type Reader ¶
type Reader interface {
io.ByteReader
io.Reader
}
Reader combines io.ByteReader and io.Reader. It's what we need to read
type Schema ¶
type Schema struct {
Type string
Object *SchemaObject
Union []Schema
}
Schema is a representation of AVRO schema JSON. Primitive types populate Type only. UnionTypes populate Type and Union fields. All other types populate Type and a subset of Object fields.
type SchemaObject ¶
type SchemaObject struct {
Type string `json:"type"`
Name string `json:"name,omitempty"`
Namespace string `json:"namespace,omitempty"`
// Fields in a record
Fields []SchemaRecordField `json:"fields,omitempty"`
// The type of each item in an array
Items Schema `json:"items,omitempty"`
// The value types of a map (keys are strings)
Values Schema `json:"values,omitempty"`
// The size of a fixed type
Size int `json:"size,omitempty"`
// The values of an enum
Symbols []string `json:"symbols,omitempty"`
}
SchemaObject contains all the fields of more complex schema types
type SchemaRecordField ¶
type SchemaRecordField struct {
Name string `json:"name,omitempty"`
Type Schema `json:"type,omitempty"`
}
SchemaRecordField represents one field of a Record schema
type StringCodec ¶
type StringCodec struct{}
StringCodec is a decoder for strings
func (StringCodec) New ¶
func (StringCodec) New() unsafe.Pointer
func (StringCodec) Skip ¶
func (StringCodec) Skip(r Reader) error
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
Package null contains avro decoders for the types in github.com/unravelin/null.
|
Package null contains avro decoders for the types in github.com/unravelin/null. |
|
Package time contains avro decoders for time.Time.
|
Package time contains avro decoders for time.Time. |