avro

package module
v0.2.2-0...-3d81695 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2020 License: Apache-2.0 Imports: 13 Imported by: 0

README

Apache Avro for Golang

Build Status

Please note that this project is still an early alpha and is subject to change.

Installation is as easy as follows:

go get github.com/elodina/go-avro

Some usage examples are located in examples folder:

go-avro now also supports code generation from Avro schemas which is available in codegen folder

Documentation

Index

Constants

View Source 
const (
	// Record schema type constant
	Record int = iota

	// Enum schema type constant
	Enum

	// Array schema type constant
	Array

	// Map schema type constant
	Map

	// Union schema type constant
	Union

	// Fixed schema type constant
	Fixed

	// String schema type constant
	String

	// Bytes schema type constant
	Bytes

	// Int schema type constant
	Int

	// Long schema type constant
	Long

	// Float schema type constant
	Float

	// Double schema type constant
	Double

	// Boolean schema type constant
	Boolean

	// Null schema type constant
	Null

	// Recursive schema type constant. Recursive is an artificial type that means a Record schema without its definition
	// that should be looked up in some registry.
	Recursive
)

Variables

View Source 
var BlockNotFinished = errors.New("Block read is unfinished")

Happens when trying to read next block without finishing the previous one.

View Source 
var EOF = errors.New("End of file reached")

Signals that an end of file or stream has been reached unexpectedly.

View Source 
var FieldDoesNotExist = errors.New("Field does not exist")

FieldDoesNotExist happens when a struct does not have a necessary field.

View Source 
var IntOverflow = errors.New("Overflowed an int value")

Happens when the given value to decode overflows maximum int32 value.

View Source 
var InvalidBool = errors.New("Invalid bool value")

Happens when given value to decode as bool is neither 0x00 nor 0x01.

View Source 
var InvalidFixedSize = errors.New("Invalid Fixed type size")

Happens when avro schema contains invalid value for fixed size.

View Source 
var InvalidInt = errors.New("Invalid int value")

Happens when given value to decode as a int is invalid

View Source 
var InvalidLong = errors.New("Invalid long value")

Happens when given value to decode as a long is invalid

View Source 
var InvalidSchema = errors.New("Invalid schema")

Happens when avro schema is unparsable or is invalid in any other way.

View Source 
var InvalidStringLength = errors.New("Invalid string length")

Happens when given value to decode as string has either negative or undecodable length.

View Source 
var InvalidSync = errors.New("Invalid sync")

Happens when file header's sync and block's sync do not match - indicates corrupted data.

View Source 
var InvalidValueType = errors.New("Invalid array or map value type")

Happens when avro schema contains invalid value for map value type or array item type.

View Source 
var LongOverflow = errors.New("Overflowed a long value")

Happens when the given value to decode overflows maximum int64 value.

View Source 
var NegativeBytesLength = errors.New("Negative bytes length")

Happens when given value to decode as bytes has negative length.

View Source 
var NestedUnionsNotAllowed = errors.New("Nested unions are not allowed")

Happens when avro schema contains a union within union.

View Source 
var NotAvroFile = errors.New("Not an Avro data file")

Indicates the given file to decode does not correspond to Avro data file format.

View Source 
var SchemaNotSet = errors.New("Schema not set")

Happens when a datum reader has no set schema.

View Source 
var UnionTypeOverflow = errors.New("Union type overflow")

UnionTypeOverflow happens when the numeric index of the union type is invalid.

Functions

func GetFullName 

func GetFullName(schema Schema) string

GetFullName returns a fully-qualified name for a schema if possible. The format is namespace.name.

func LoadSchemas 

func LoadSchemas(path string) map[string]Schema

LoadSchemas loads and parses a schema file or directory. Directory names MUST end with "/"

Types

type ArraySchema 

type ArraySchema struct {
	Items      Schema
	Properties map[string]interface{}
}

ArraySchema implements Schema and represents Avro array type.

func (*ArraySchema) GetName 

func (*ArraySchema) GetName() string

GetName returns a type name for this ArraySchema.

func (*ArraySchema) MarshalJSON 

func (s *ArraySchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*ArraySchema) Prop 

func (s *ArraySchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*ArraySchema) String 

func (s *ArraySchema) String() string

String returns a JSON representation of ArraySchema.

func (*ArraySchema) Type 

func (*ArraySchema) Type() int

Type returns a type constant for this ArraySchema.

func (*ArraySchema) Validate 

func (s *ArraySchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type AvroRecord 

type AvroRecord interface {
	// Schema returns an Avro schema for this AvroRecord.
	Schema() Schema
}

AvroRecord is an interface for anything that has an Avro schema and can be serialized/deserialized by this library.

type BinaryDecoder 

type BinaryDecoder struct {
	// contains filtered or unexported fields
}

BinaryDecoder implements Decoder and provides low-level support for deserializing Avro values.

func NewBinaryDecoder 

func NewBinaryDecoder(buf []byte) *BinaryDecoder

NewBinaryDecoder creates a new BinaryDecoder to read from a given buffer.

func (*BinaryDecoder) ArrayNext 

func (bd *BinaryDecoder) ArrayNext() (int64, error)

ArrayNext processes the next block of an array and returns the number of items in the block. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) MapNext 

func (bd *BinaryDecoder) MapNext() (int64, error)

MapNext processes the next block of map entries and returns the number of items in the block. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadArrayStart 

func (bd *BinaryDecoder) ReadArrayStart() (int64, error)

ReadArrayStart reads and returns the size of the first block of an array. If call to this return non-zero, then the caller should read the indicated number of items and then call ArrayNext() to find out the number of items in the next block. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadBoolean 

func (bd *BinaryDecoder) ReadBoolean() (bool, error)

ReadBoolean reads a boolean value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadBytes 

func (bd *BinaryDecoder) ReadBytes() ([]byte, error)

ReadBytes reads a bytes value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadDouble 

func (bd *BinaryDecoder) ReadDouble() (float64, error)

ReadDouble reads a double value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadEnum 

func (bd *BinaryDecoder) ReadEnum() (int32, error)

ReadEnum reads an enum value (which is an Avro int value). Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadFixed 

func (bd *BinaryDecoder) ReadFixed(bytes []byte) error

ReadFixed reads fixed sized binary object into the provided buffer. Returns an error if it occurs.

func (*BinaryDecoder) ReadFixedWithBounds 

func (bd *BinaryDecoder) ReadFixedWithBounds(bytes []byte, start int, length int) error

ReadFixedWithBounds reads fixed sized binary object into the provided buffer. The second parameter is the position where the data needs to be written, the third is the size of binary object. Returns an error if it occurs.

func (*BinaryDecoder) ReadFloat 

func (bd *BinaryDecoder) ReadFloat() (float32, error)

ReadFloat reads a float value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadInt 

func (bd *BinaryDecoder) ReadInt() (int32, error)

ReadInt reads an int value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadLong 

func (bd *BinaryDecoder) ReadLong() (int64, error)

ReadLong reads a long value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadMapStart 

func (bd *BinaryDecoder) ReadMapStart() (int64, error)

ReadMapStart reads and returns the size of the first block of map entries. If call to this return non-zero, then the caller should read the indicated number of items and then call MapNext() to find out the number of items in the next block. Usage is similar to ReadArrayStart(). Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadNull 

func (bd *BinaryDecoder) ReadNull() (interface{}, error)

ReadNull reads a null value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) ReadString 

func (bd *BinaryDecoder) ReadString() (string, error)

ReadString reads a string value. Returns a decoded value and an error if it occurs.

func (*BinaryDecoder) Seek 

func (bd *BinaryDecoder) Seek(pos int64)

Seek sets the reading position of this Decoder to a given value allowing to skip items etc.

func (*BinaryDecoder) SetBlock 

func (bd *BinaryDecoder) SetBlock(block *DataBlock)

SetBlock is used for Avro Object Container Files where the data is split in blocks and sets a data block for this decoder and sets the position to the start of this block.

func (*BinaryDecoder) Tell 

func (bd *BinaryDecoder) Tell() int64

Tell returns the current reading position of this Decoder.

type BinaryEncoder 

type BinaryEncoder struct {
	// contains filtered or unexported fields
}

BinaryEncoder implements Encoder and provides low-level support for serializing Avro values.

func NewBinaryEncoder 

func NewBinaryEncoder(buffer io.Writer) *BinaryEncoder

NewBinaryEncoder creates a new BinaryEncoder that will write to a given io.Writer.

func (*BinaryEncoder) WriteArrayNext 

func (be *BinaryEncoder) WriteArrayNext(count int64)

WriteArrayNext should be called after finishing writing an array block either passing it the number of items in next block or 0 indicating the end of array.

func (*BinaryEncoder) WriteArrayStart 

func (be *BinaryEncoder) WriteArrayStart(count int64)

WriteArrayStart should be called when starting to serialize an array providing it with a number of items in array block.

func (*BinaryEncoder) WriteBoolean 

func (be *BinaryEncoder) WriteBoolean(x bool)

WriteBoolean writes a boolean value.

func (*BinaryEncoder) WriteBytes 

func (be *BinaryEncoder) WriteBytes(x []byte)

WriteBytes writes a bytes value.

func (*BinaryEncoder) WriteDouble 

func (be *BinaryEncoder) WriteDouble(x float64)

WriteDouble writes a double value.

func (*BinaryEncoder) WriteFloat 

func (be *BinaryEncoder) WriteFloat(x float32)

WriteFloat writes a float value.

func (*BinaryEncoder) WriteInt 

func (be *BinaryEncoder) WriteInt(x int32)

WriteInt writes an int value.

func (*BinaryEncoder) WriteLong 

func (be *BinaryEncoder) WriteLong(x int64)

WriteLong writes a long value.

func (*BinaryEncoder) WriteMapNext 

func (be *BinaryEncoder) WriteMapNext(count int64)

WriteMapNext should be called after finishing writing a map block either passing it the number of items in next block or 0 indicating the end of map.

func (*BinaryEncoder) WriteMapStart 

func (be *BinaryEncoder) WriteMapStart(count int64)

WriteMapStart should be called when starting to serialize a map providing it with a number of items in map block.

func (*BinaryEncoder) WriteNull 

func (be *BinaryEncoder) WriteNull(_ interface{})

WriteNull writes a null value. Doesn't actually do anything in this implementation.

func (*BinaryEncoder) WriteRaw 

func (be *BinaryEncoder) WriteRaw(x []byte)

WriteRaw writes raw bytes to this Encoder.

func (*BinaryEncoder) WriteString 

func (be *BinaryEncoder) WriteString(x string)

WriteString writes a string value.

type BooleanSchema 

type BooleanSchema struct{}

BooleanSchema implements Schema and represents Avro boolean type.

func (*BooleanSchema) GetName 

func (*BooleanSchema) GetName() string

GetName returns a type name for this BooleanSchema.

func (*BooleanSchema) MarshalJSON 

func (*BooleanSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BooleanSchema) Prop 

func (*BooleanSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BooleanSchema.

func (*BooleanSchema) String 

func (*BooleanSchema) String() string

String returns a JSON representation of BooleanSchema.

func (*BooleanSchema) Type 

func (*BooleanSchema) Type() int

Type returns a type constant for this BooleanSchema.

func (*BooleanSchema) Validate 

func (*BooleanSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type BytesSchema 

type BytesSchema struct{}

BytesSchema implements Schema and represents Avro bytes type.

func (*BytesSchema) GetName 

func (*BytesSchema) GetName() string

GetName returns a type name for this BytesSchema.

func (*BytesSchema) MarshalJSON 

func (*BytesSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BytesSchema) Prop 

func (*BytesSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BytesSchema.

func (*BytesSchema) String 

func (*BytesSchema) String() string

String returns a JSON representation of BytesSchema.

func (*BytesSchema) Type 

func (*BytesSchema) Type() int

Type returns a type constant for this BytesSchema.

func (*BytesSchema) Validate 

func (*BytesSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type CodeGenerator 

type CodeGenerator struct {
	// contains filtered or unexported fields
}

CodeGenerator is a code generation tool for structs from given Avro schemas.

func NewCodeGenerator 

func NewCodeGenerator(schemas []string) *CodeGenerator

NewCodeGenerator creates a new CodeGenerator for given Avro schemas.

func (*CodeGenerator) Generate 

func (codegen *CodeGenerator) Generate() (string, error)

Generate generates source code for Avro schemas specified on creation. The ouput is Go formatted source code that contains struct definitions for all given schemas. May return an error if code generation fails, e.g. due to unparsable schema.

type DataBlock 

type DataBlock struct {
	// Actual data
	Data []byte

	// Number of entries encoded in Data.
	NumEntries int64

	// Size of data buffer in bytes.
	BlockSize int

	// Number of unread entries in this DataBlock.
	BlockRemaining int64
}

DataBlock is a structure that holds a certain amount of entries and the actual buffer to read from.

type DataFileReader 

type DataFileReader struct {
	// contains filtered or unexported fields
}

DataFileReader is a reader for Avro Object Container Files. More here: https://avro.apache.org/docs/current/spec.html#Object+Container+Files

func NewDataFileReader 

func NewDataFileReader(filename string, datumReader DatumReader) (*DataFileReader, error)

NewDataFileReader creates a new DataFileReader for a given file and using the given DatumReader to read the data from that file. May return an error if the file contains invalid data or is just missing.

func (*DataFileReader) Close 

func (reader *DataFileReader) Close() error

Close any resources opened by the DataFileReader

func (*DataFileReader) Next 

func (reader *DataFileReader) Next(v interface{}) (bool, error)

Next reads the next value from file and fills the given value with data. First return value indicates whether the read was successful. Second return value indicates whether there was an error while reading data. Returns (false, nil) when no more data left to read.

func (*DataFileReader) NextBlock 

func (reader *DataFileReader) NextBlock() error

NextBlock tells this DataFileReader to skip current block and move to next one. May return an error if the block is malformed or no more blocks left to read.

func (*DataFileReader) Seek 

func (reader *DataFileReader) Seek(pos int64)

Seek switches the reading position in this DataFileReader to a provided value.

type DataFileWriter 

type DataFileWriter struct {
	// contains filtered or unexported fields
}

DataFileWriter lets you write object container files.

func NewDataFileWriter 

func NewDataFileWriter(output io.Writer, schema Schema, datumWriter DatumWriter) (writer *DataFileWriter, err error)

NewDataFileWriter creates a new DataFileWriter for given output and schema using the given DatumWriter to write the data to that Writer. May return an error if writing fails.

func (*DataFileWriter) Close 

func (w *DataFileWriter) Close() error

Close this DataFileWriter. This is required to finish out the data file format. After Close() is called, this DataFileWriter cannot be used anymore.

func (*DataFileWriter) Flush 

func (w *DataFileWriter) Flush() error

Flush out any previously written datums to our underlying io.Writer. Does nothing if no datums had previously been written.

It's up to the library user to decide how often to flush; doing it often will spend a lot of time on tiny I/O but save memory.

func (*DataFileWriter) Write 

func (w *DataFileWriter) Write(v interface{}) error

Write out a single datum.

Encoded datums are buffered internally and will not be written to the underlying io.Writer until Flush() is called.

type DatumReader 

type DatumReader interface {
	// Reads a single structured entry using this DatumReader according to provided Schema.
	// Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type.
	// May return an error indicating a read failure.
	Read(interface{}, Decoder) error

	// Sets the schema for this DatumReader to know the data structure.
	// Note that it must be called before calling Read.
	SetSchema(Schema)
}

DatumReader is an interface that is responsible for reading structured data according to schema from a decoder

type DatumWriter 

type DatumWriter interface {
	// Write writes a single entry using this DatumWriter according to provided Schema.
	// Accepts a value to write and Encoder to write to.
	// May return an error indicating a write failure.
	Write(interface{}, Encoder) error

	// Sets the schema for this DatumWriter to know the data structure.
	// Note that it must be called before calling Write.
	SetSchema(Schema)
}

DatumWriter is an interface that is responsible for writing structured data according to schema to an encoder.

type Decoder 

type Decoder interface {
	// Reads a null value. Returns a decoded value and an error if it occurs.
	ReadNull() (interface{}, error)

	// Reads a boolean value. Returns a decoded value and an error if it occurs.
	ReadBoolean() (bool, error)

	// Reads an in value. Returns a decoded value and an error if it occurs.
	ReadInt() (int32, error)

	// Reads a long value. Returns a decoded value and an error if it occurs.
	ReadLong() (int64, error)

	// Reads a float value. Returns a decoded value and an error if it occurs.
	ReadFloat() (float32, error)

	// Reads a double value. Returns a decoded value and an error if it occurs.
	ReadDouble() (float64, error)

	// Reads a bytes value. Returns a decoded value and an error if it occurs.
	ReadBytes() ([]byte, error)

	// Reads a string value. Returns a decoded value and an error if it occurs.
	ReadString() (string, error)

	// Reads an enum value (which is an Avro int value). Returns a decoded value and an error if it occurs.
	ReadEnum() (int32, error)

	// Reads and returns the size of the first block of an array. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call ArrayNext() to find out the number of items in the
	// next block. Returns a decoded value and an error if it occurs.
	ReadArrayStart() (int64, error)

	// Processes the next block of an array and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	ArrayNext() (int64, error)

	// Reads and returns the size of the first block of map entries. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call MapNext() to find out the number of items in the
	// next block. Usage is similar to ReadArrayStart(). Returns a decoded value and an error if it occurs.
	ReadMapStart() (int64, error)

	// Processes the next block of map entries and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	MapNext() (int64, error)

	// Reads fixed sized binary object into the provided buffer.
	// Returns an error if it occurs.
	ReadFixed([]byte) error

	// Reads fixed sized binary object into the provided buffer.
	// The second parameter is the position where the data needs to be written, the third is the size of binary object.
	// Returns an error if it occurs.
	ReadFixedWithBounds([]byte, int, int) error

	// SetBlock is used for Avro Object Container Files where the data is split in blocks and sets a data block
	// for this decoder and sets the position to the start of this block.
	SetBlock(*DataBlock)

	// Seek sets the reading position of this Decoder to a given value allowing to skip items etc.
	Seek(int64)

	// Tell returns the current reading position of this Decoder.
	Tell() int64
}

Decoder is an interface that provides low-level support for deserializing Avro values.

type DoubleSchema 

type DoubleSchema struct{}

DoubleSchema implements Schema and represents Avro double type.

func (*DoubleSchema) GetName 

func (*DoubleSchema) GetName() string

GetName returns a type name for this DoubleSchema.

func (*DoubleSchema) MarshalJSON 

func (*DoubleSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*DoubleSchema) Prop 

func (*DoubleSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for DoubleSchema.

func (*DoubleSchema) String 

func (*DoubleSchema) String() string

Returns a JSON representation of DoubleSchema.

func (*DoubleSchema) Type 

func (*DoubleSchema) Type() int

Type returns a type constant for this DoubleSchema.

func (*DoubleSchema) Validate 

func (*DoubleSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Encoder 

type Encoder interface {
	// Writes a null value. Doesn't actually do anything but may advance the state of Encoder implementation if it
	// is stateful.
	WriteNull(interface{})

	// Writes a boolean value.
	WriteBoolean(bool)

	// Writes an int value.
	WriteInt(int32)

	// Writes a long value.
	WriteLong(int64)

	// Writes a float value.
	WriteFloat(float32)

	// Writes a double value.
	WriteDouble(float64)

	// Writes a bytes value.
	WriteBytes([]byte)

	// Writes a string value.
	WriteString(string)

	// WriteArrayStart should be called when starting to serialize an array providing it with a number of items in
	// array block.
	WriteArrayStart(int64)

	// WriteArrayNext should be called after finishing writing an array block either passing it the number of items in
	// next block or 0 indicating the end of array.
	WriteArrayNext(int64)

	// WriteMapStart should be called when starting to serialize a map providing it with a number of items in
	// map block.
	WriteMapStart(int64)

	// WriteMapNext should be called after finishing writing a map block either passing it the number of items in
	// next block or 0 indicating the end of map.
	WriteMapNext(int64)

	// Writes raw bytes to this Encoder.
	WriteRaw([]byte)
}

Encoder is an interface that provides low-level support for serializing Avro values.

type EnumSchema 

type EnumSchema struct {
	Name       string
	Namespace  string
	Aliases    []string
	Doc        string
	Symbols    []string
	Properties map[string]interface{}
}

EnumSchema implements Schema and represents Avro enum type.

func (*EnumSchema) GetName 

func (s *EnumSchema) GetName() string

GetName returns an enum name for this EnumSchema.

func (*EnumSchema) MarshalJSON 

func (s *EnumSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*EnumSchema) Prop 

func (s *EnumSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*EnumSchema) String 

func (s *EnumSchema) String() string

String returns a JSON representation of EnumSchema.

func (*EnumSchema) Type 

func (*EnumSchema) Type() int

Type returns a type constant for this EnumSchema.

func (*EnumSchema) Validate 

func (*EnumSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FixedSchema 

type FixedSchema struct {
	Namespace  string
	Name       string
	Size       int
	Properties map[string]interface{}
}

FixedSchema implements Schema and represents Avro fixed type.

func (*FixedSchema) GetName 

func (s *FixedSchema) GetName() string

GetName returns a fixed name for this FixedSchema.

func (*FixedSchema) MarshalJSON 

func (s *FixedSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*FixedSchema) Prop 

func (s *FixedSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*FixedSchema) String 

func (s *FixedSchema) String() string

String returns a JSON representation of FixedSchema.

func (*FixedSchema) Type 

func (*FixedSchema) Type() int

Type returns a type constant for this FixedSchema.

func (*FixedSchema) Validate 

func (s *FixedSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FloatSchema 

type FloatSchema struct{}

FloatSchema implements Schema and represents Avro float type.

func (*FloatSchema) GetName 

func (*FloatSchema) GetName() string

GetName returns a type name for this FloatSchema.

func (*FloatSchema) MarshalJSON 

func (*FloatSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*FloatSchema) Prop 

func (*FloatSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for FloatSchema.

func (*FloatSchema) String 

func (*FloatSchema) String() string

String returns a JSON representation of FloatSchema.

func (*FloatSchema) Type 

func (*FloatSchema) Type() int

Type returns a type constant for this FloatSchema.

func (*FloatSchema) Validate 

func (*FloatSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type GenericDatumReader 

type GenericDatumReader struct {
	// contains filtered or unexported fields
}

GenericDatumReader implements DatumReader and is used for filling GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) with data. Each value passed to Read is expected to be a pointer.

func NewGenericDatumReader 

func NewGenericDatumReader() *GenericDatumReader

NewGenericDatumReader creates a new GenericDatumReader.

func (*GenericDatumReader) Read 

func (reader *GenericDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single entry using this GenericDatumReader. Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type. May return an error indicating a read failure.

func (*GenericDatumReader) SetSchema 

func (reader *GenericDatumReader) SetSchema(schema Schema)

SetSchema sets the schema for this GenericDatumReader to know the data structure. Note that it must be called before calling Read.

type GenericDatumWriter 

type GenericDatumWriter struct {
	// contains filtered or unexported fields
}

GenericDatumWriter implements DatumWriter and is used for writing GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) to a given Encoder.

func NewGenericDatumWriter 

func NewGenericDatumWriter() *GenericDatumWriter

NewGenericDatumWriter creates a new GenericDatumWriter.

func (*GenericDatumWriter) SetSchema 

func (writer *GenericDatumWriter) SetSchema(schema Schema)

SetSchema sets the provided schema for this GenericDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*GenericDatumWriter) Write 

func (writer *GenericDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single entry using this GenericDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. May return an error indicating a write failure.

type GenericEnum 

type GenericEnum struct {
	// Avro enum symbols.
	Symbols []string
	// contains filtered or unexported fields
}

GenericEnum is a generic Avro enum representation. This is still subject to change and may be rethought.

func NewGenericEnum 

func NewGenericEnum(symbols []string) *GenericEnum

NewGenericEnum returns a new GenericEnum that uses provided enum symbols.

func (*GenericEnum) Get 

func (enum *GenericEnum) Get() string

Get gets the string value for this enum (e.g. symbol).

func (*GenericEnum) GetIndex 

func (enum *GenericEnum) GetIndex() int32

GetIndex gets the numeric value for this enum.

func (*GenericEnum) Set 

func (enum *GenericEnum) Set(symbol string)

Set sets the string value for this enum (e.g. symbol). Panics if the given symbol does not exist in this enum.

func (*GenericEnum) SetIndex 

func (enum *GenericEnum) SetIndex(index int32)

SetIndex sets the numeric value for this enum.

type GenericRecord 

type GenericRecord struct {
	// contains filtered or unexported fields
}

GenericRecord is a generic instance of a record schema. Fields are accessible by their name.

func NewGenericRecord 

func NewGenericRecord(schema Schema) *GenericRecord

NewGenericRecord creates a new GenericRecord.

func (*GenericRecord) Get 

func (gr *GenericRecord) Get(name string) interface{}

Get gets a value by its name.

func (*GenericRecord) Map 

func (gr *GenericRecord) Map() map[string]interface{}

Map returns a map representation of this GenericRecord.

func (*GenericRecord) Schema 

func (gr *GenericRecord) Schema() Schema

Schema returns a schema for this GenericRecord.

func (*GenericRecord) Set 

func (gr *GenericRecord) Set(name string, value interface{})

Set sets a value for a given name.

func (*GenericRecord) String 

func (gr *GenericRecord) String() string

String returns a JSON representation of this GenericRecord.

type IntSchema 

type IntSchema struct{}

IntSchema implements Schema and represents Avro int type.

func (*IntSchema) GetName 

func (*IntSchema) GetName() string

GetName returns a type name for this IntSchema.

func (*IntSchema) MarshalJSON 

func (*IntSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*IntSchema) Prop 

func (*IntSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for IntSchema.

func (*IntSchema) String 

func (*IntSchema) String() string

String returns a JSON representation of IntSchema.

func (*IntSchema) Type 

func (*IntSchema) Type() int

Type returns a type constant for this IntSchema.

func (*IntSchema) Validate 

func (*IntSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type LongSchema 

type LongSchema struct{}

LongSchema implements Schema and represents Avro long type.

func (*LongSchema) GetName 

func (*LongSchema) GetName() string

GetName returns a type name for this LongSchema.

func (*LongSchema) MarshalJSON 

func (*LongSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*LongSchema) Prop 

func (*LongSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for LongSchema.

func (*LongSchema) String 

func (*LongSchema) String() string

Returns a JSON representation of LongSchema.

func (*LongSchema) Type 

func (*LongSchema) Type() int

Type returns a type constant for this LongSchema.

func (*LongSchema) Validate 

func (*LongSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type MapSchema 

type MapSchema struct {
	Values     Schema
	Properties map[string]interface{}
}

MapSchema implements Schema and represents Avro map type.

func (*MapSchema) GetName 

func (*MapSchema) GetName() string

GetName returns a type name for this MapSchema.

func (*MapSchema) MarshalJSON 

func (s *MapSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*MapSchema) Prop 

func (s *MapSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*MapSchema) String 

func (s *MapSchema) String() string

String returns a JSON representation of MapSchema.

func (*MapSchema) Type 

func (*MapSchema) Type() int

Type returns a type constant for this MapSchema.

func (*MapSchema) Validate 

func (s *MapSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type NullSchema 

type NullSchema struct{}

NullSchema implements Schema and represents Avro null type.

func (*NullSchema) GetName 

func (*NullSchema) GetName() string

GetName returns a type name for this NullSchema.

func (*NullSchema) MarshalJSON 

func (*NullSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*NullSchema) Prop 

func (*NullSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for NullSchema.

func (*NullSchema) String 

func (*NullSchema) String() string

String returns a JSON representation of NullSchema.

func (*NullSchema) Type 

func (*NullSchema) Type() int

Type returns a type constant for this NullSchema.

func (*NullSchema) Validate 

func (*NullSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Reader 

type Reader interface {
	Read(dec Decoder) error
}

Reader is an interface that may be implemented to avoid using runtime reflection during deserialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

type RecordSchema 

type RecordSchema struct {
	Name       string   `json:"name,omitempty"`
	Namespace  string   `json:"namespace,omitempty"`
	Doc        string   `json:"doc,omitempty"`
	Aliases    []string `json:"aliases,omitempty"`
	Properties map[string]interface{}
	Fields     []*SchemaField `json:"fields"`
}

RecordSchema implements Schema and represents Avro record type.

func (*RecordSchema) GetName 

func (s *RecordSchema) GetName() string

GetName returns a record name for this RecordSchema.

func (*RecordSchema) MarshalJSON 

func (s *RecordSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*RecordSchema) Prop 

func (s *RecordSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*RecordSchema) String 

func (s *RecordSchema) String() string

String returns a JSON representation of RecordSchema.

func (*RecordSchema) Type 

func (*RecordSchema) Type() int

Type returns a type constant for this RecordSchema.

func (*RecordSchema) Validate 

func (s *RecordSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type RecursiveSchema 

type RecursiveSchema struct {
	Actual *RecordSchema
}

RecursiveSchema implements Schema and represents Avro record type without a definition (e.g. that should be looked up).

func (*RecursiveSchema) GetName 

func (s *RecursiveSchema) GetName() string

GetName returns a record name for enclosed RecordSchema.

func (*RecursiveSchema) MarshalJSON 

func (s *RecursiveSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*RecursiveSchema) Prop 

func (*RecursiveSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for RecursiveSchema.

func (*RecursiveSchema) String 

func (s *RecursiveSchema) String() string

String returns a JSON representation of RecursiveSchema.

func (*RecursiveSchema) Type 

func (*RecursiveSchema) Type() int

Type returns a type constant for this RecursiveSchema.

func (*RecursiveSchema) Validate 

func (s *RecursiveSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Schema 

type Schema interface {
	// Returns an integer constant representing this schema type.
	Type() int

	// If this is a record, enum or fixed, returns its name, otherwise the name of the primitive type.
	GetName() string

	// Gets a custom non-reserved property from this schema and a bool representing if it exists.
	Prop(key string) (interface{}, bool)

	// Converts this schema to its JSON representation.
	String() string

	// Checks whether the given value is writeable to this schema.
	Validate(v reflect.Value) bool
}

Schema is an interface representing a single Avro schema (both primitive and complex).

func MustParseSchema 

func MustParseSchema(rawSchema string) Schema

MustParseSchema is like ParseSchema, but panics if the given schema cannot be parsed.

func ParseSchema 

func ParseSchema(rawSchema string) (Schema, error)

ParseSchema parses a given schema without provided schemas to reuse. Equivalent to call ParseSchemaWithResistry(rawSchema, make(map[string]Schema)) May return an error if schema is not parsable or has insufficient information about any type.

func ParseSchemaFile 

func ParseSchemaFile(file string) (Schema, error)

ParseSchemaFile parses a given file. May return an error if schema is not parsable or file does not exist.

func ParseSchemaWithRegistry 

func ParseSchemaWithRegistry(rawSchema string, schemas map[string]Schema) (Schema, error)

ParseSchemaWithRegistry parses a given schema using the provided registry for type lookup. Registry will be filled up during parsing. May return an error if schema is not parsable or has insufficient information about any type.

func Prepare 

func Prepare(schema Schema) Schema

Prepare optimizes a schema for decoding/encoding.

It makes a recursive copy of the schema given and returns an immutable wrapper of the schema with some optimizations applied.

func PrepareResolving 

func PrepareResolving(writer, reader Schema) Schema

PrepareResolving handles reading where the writer and reader schemas differ but are compatible. Support is currently limited to adding or removing record fields.

type SchemaField 

type SchemaField struct {
	Name       string      `json:"name,omitempty"`
	Doc        string      `json:"doc,omitempty"`
	Default    interface{} `json:"default"`
	Type       Schema      `json:"type,omitempty"`
	Properties map[string]interface{}
}

SchemaField represents a schema field for Avro record.

func (*SchemaField) MarshalJSON 

func (s *SchemaField) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema field as JSON.

func (*SchemaField) Prop 

func (this *SchemaField) Prop(key string) (interface{}, bool)

Gets a custom non-reserved property from this schemafield and a bool representing if it exists.

func (*SchemaField) String 

func (s *SchemaField) String() string

String returns a JSON representation of SchemaField.

type SpecificDatumReader 

type SpecificDatumReader struct {
	// contains filtered or unexported fields
}

SpecificDatumReader implements DatumReader and is used for filling Go structs with data. Each value passed to Read is expected to be a pointer.

func NewSpecificDatumReader 

func NewSpecificDatumReader() *SpecificDatumReader

NewSpecificDatumReader creates a new SpecificDatumReader.

func (*SpecificDatumReader) Read 

func (reader *SpecificDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single structured entry using this SpecificDatumReader. Accepts a Go struct with exported fields to fill with data and a Decoder to read from. Given value MUST be of pointer type. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a read failure.

func (*SpecificDatumReader) SetSchema 

func (reader *SpecificDatumReader) SetSchema(schema Schema)

SetSchema sets the schema for this SpecificDatumReader to know the data structure. Note that it must be called before calling Read.

type SpecificDatumWriter 

type SpecificDatumWriter struct {
	// contains filtered or unexported fields
}

SpecificDatumWriter implements DatumWriter and is used for writing Go structs in Avro format.

func NewSpecificDatumWriter 

func NewSpecificDatumWriter() *SpecificDatumWriter

NewSpecificDatumWriter creates a new SpecificDatumWriter.

func (*SpecificDatumWriter) SetSchema 

func (writer *SpecificDatumWriter) SetSchema(schema Schema)

SetSchema sets the provided schema for this SpecificDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*SpecificDatumWriter) Write 

func (writer *SpecificDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single Go struct using this SpecificDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a write failure.

type StringSchema 

type StringSchema struct{}

StringSchema implements Schema and represents Avro string type.

func (*StringSchema) GetName 

func (*StringSchema) GetName() string

GetName returns a type name for this StringSchema.

func (*StringSchema) MarshalJSON 

func (*StringSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*StringSchema) Prop 

func (*StringSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for StringSchema.

func (*StringSchema) String 

func (*StringSchema) String() string

Returns a JSON representation of StringSchema.

func (*StringSchema) Type 

func (*StringSchema) Type() int

Type returns a type constant for this StringSchema.

func (*StringSchema) Validate 

func (*StringSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type UnionSchema 

type UnionSchema struct {
	Types []Schema
}

UnionSchema implements Schema and represents Avro union type.

func (*UnionSchema) GetName 

func (*UnionSchema) GetName() string

GetName returns a type name for this UnionSchema.

func (*UnionSchema) GetType 

func (s *UnionSchema) GetType(v reflect.Value) int

GetType gets the index of actual union type for a given value.

func (*UnionSchema) MarshalJSON 

func (s *UnionSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*UnionSchema) Prop 

func (*UnionSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for UnionSchema.

func (*UnionSchema) String 

func (s *UnionSchema) String() string

String returns a JSON representation of UnionSchema.

func (*UnionSchema) Type 

func (*UnionSchema) Type() int

Type returns a type constant for this UnionSchema.

func (*UnionSchema) Validate 

func (s *UnionSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Writer 

type Writer interface {
	Write(enc Encoder) error
}

Writer is an interface that may be implemented to avoid using runtime reflection during serialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

Source Files

View all Source files

Directories

Path Synopsis
examples
data_file
generic_datum
load_schema
specific_datum

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.