avro

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 10, 2023 License: Apache-2.0 Imports: 15 Imported by: 0

README

Apache Avro for Golang

Build Status GoDoc

Support for decoding/encoding avro using both map-style access (GenericRecord) and to/from arbitrary Go structs (SpecificRecord).

This library started as a fork of elodina/go-avro but has now proceeded to become a maintained library.

Installation

Installation via go get:

go get github.com/kjuulh/go-avro

Documentation

Some usage examples are located in examples folder:

About This fork

This fork separated from elodina/go-avro in December 2016 because of the project not responding to PR's since around May 2016. Had tried to contact them to get maintainer access but the original maintainer no longer is able to make those changes.

Originally, we were waiting in hope the elodina maintainer would return, but it hasn't happened, so the plan now is to proceed with this as its own library and take PRs, push for feature additions and version bumps.

Documentation

Overview

Package avro encodes/decodes avro schemas to your struct or a map.

Overview

Go-avro parses .avsc schemas from files and then lets you work with them. 

schema, err := avro.ParseSchemaFile("person.avsc")
// important: handle err!

Struct Mapping

When using SpecificDecoder, the implementation uses struct tags to map avro messages into your struct. This helps because it makes schema evolution easier and avoids having a ton of casting and dictionary checks in your user code.

Say you had the schema: 

{
  "type": "record",
  "name": "Person",
  "fields" : [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"}
    {"name": "location", "type": {
      "type": "record",
      "name": "Location",
      "fields": [
        {"name": "latitude", "type": "double"},
        {"name": "longitude", "type": "double"}
      ]
    }}
  ]
}

This could be mapped to a SpecificRecord by using structs: 

type Person struct {
	Id       int32     `avro:"id"`
	Name     string    `avro:"name"`
	Location *Location `avro:"location"`
}

type Location struct {
	Latitude  float64
	Longitude float64
}

If the `avro:` struct tag is omitted, the default mapping lower-cases the first letter only. It's better to just explicitly define where possible.

Mapped types:

  • avro 'int' is always 32-bit, so maps to golang 'int32'
  • avro 'long' is always mapped to 'int64'
  • avro 'float' -> float32
  • avro 'double' -> 'float64'
  • most other ones are obvious

Type unions are a bit more tricky. For a complex type union, the only valid mapping is interface{}. However, for a type union with only "null" and one other type (very typical) you can map it as a pointer type and keep type safety.

Index

Examples

Constants

View Source 
const (
	// Record schema type constant
	Record int = iota

	// Enum schema type constant
	Enum

	// Array schema type constant
	Array

	// Map schema type constant
	Map

	// Union schema type constant
	Union

	// Fixed schema type constant
	Fixed

	// String schema type constant
	String

	// Bytes schema type constant
	Bytes

	// Int schema type constant
	Int

	// Long schema type constant
	Long

	// Float schema type constant
	Float

	// Double schema type constant
	Double

	// Boolean schema type constant
	Boolean

	// Null schema type constant
	Null

	// Recursive schema type constant. Recursive is an artificial type that means a Record schema without its definition
	// that should be looked up in some registry.
	Recursive
)

Variables

View Source 
var ErrBlockNotFinished = errors.New("Block read is unfinished")

Happens when trying to read next block without finishing the previous one.

View Source 
var ErrIntOverflow = errors.New("Overflowed an int value")

Happens when the given value to decode overflows maximum int32 value.

View Source 
var ErrInvalidBool = errors.New("Invalid bool value")

Happens when given value to decode as bool is neither 0x00 nor 0x01.

View Source 
var ErrInvalidFixedSize = errors.New("Invalid Fixed type size")

Happens when avro schema contains invalid value for fixed size.

View Source 
var ErrInvalidInt = errors.New("Invalid int value")

Happens when given value to decode as a int is invalid

View Source 
var ErrInvalidLong = errors.New("Invalid long value")

Happens when given value to decode as a long is invalid

View Source 
var ErrInvalidSchema = errors.New("Invalid schema")

Happens when avro schema is unparsable or is invalid in any other way.

View Source 
var ErrInvalidStringLength = errors.New("Invalid string length")

Happens when given value to decode as string has either negative or undecodable length.

View Source 
var ErrLongOverflow = errors.New("Overflowed a long value")

Happens when the given value to decode overflows maximum int64 value.

View Source 
var ErrNegativeBytesLength = errors.New("Negative bytes length")

Happens when given value to decode as bytes has negative length.

View Source 
var ErrNotAvroFile = errors.New("Not an Avro data file")

Indicates the given file to decode does not correspond to Avro data file format.

View Source 
var ErrSchemaNotSet = errors.New("Schema not set")

Happens when a datum reader has no set schema.

View Source 
var ErrUnexpectedEOF = io.ErrUnexpectedEOF

Signals that an end of file or stream has been reached unexpectedly.

View Source 
var ErrUnionTypeOverflow = errors.New("Union type overflow")

UnionTypeOverflow happens when the numeric index of the union type is invalid.

Functions

func GetFullName 

func GetFullName(schema Schema) string

GetFullName returns a fully-qualified name for a schema if possible. The format is namespace.name.

func LoadSchemas 

func LoadSchemas(path string) map[string]Schema

LoadSchemas loads and parses a schema file or directory. Directory names MUST end with "/"

func NewFieldDoesNotExistError 

func NewFieldDoesNotExistError(field string) error

Specify a custom error message for indicating which necessary field in the struct is missing.

Types

type ArraySchema 

type ArraySchema struct {
	Items      Schema
	Properties map[string]interface{}
}

ArraySchema implements Schema and represents Avro array type.

func (*ArraySchema) GetName 

func (*ArraySchema) GetName() string

GetName returns a type name for this ArraySchema.

func (*ArraySchema) MarshalJSON 

func (s *ArraySchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*ArraySchema) Prop 

func (s *ArraySchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*ArraySchema) String 

func (s *ArraySchema) String() string

String returns a JSON representation of ArraySchema.

func (*ArraySchema) Type 

func (*ArraySchema) Type() int

Type returns a type constant for this ArraySchema.

func (*ArraySchema) Validate 

func (s *ArraySchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type AvroRecord 

type AvroRecord interface {
	// Schema returns an Avro schema for this AvroRecord.
	Schema() Schema
}

AvroRecord is an interface for anything that has an Avro schema and can be serialized/deserialized by this library.

type BooleanSchema 

type BooleanSchema struct{}

BooleanSchema implements Schema and represents Avro boolean type.

func (*BooleanSchema) GetName 

func (*BooleanSchema) GetName() string

GetName returns a type name for this BooleanSchema.

func (*BooleanSchema) MarshalJSON 

func (*BooleanSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BooleanSchema) Prop 

func (*BooleanSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BooleanSchema.

func (*BooleanSchema) String 

func (*BooleanSchema) String() string

String returns a JSON representation of BooleanSchema.

func (*BooleanSchema) Type 

func (*BooleanSchema) Type() int

Type returns a type constant for this BooleanSchema.

func (*BooleanSchema) Validate 

func (*BooleanSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type BytesSchema 

type BytesSchema struct{}

BytesSchema implements Schema and represents Avro bytes type.

func (*BytesSchema) GetName 

func (*BytesSchema) GetName() string

GetName returns a type name for this BytesSchema.

func (*BytesSchema) MarshalJSON 

func (*BytesSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*BytesSchema) Prop 

func (*BytesSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for BytesSchema.

func (*BytesSchema) String 

func (*BytesSchema) String() string

String returns a JSON representation of BytesSchema.

func (*BytesSchema) Type 

func (*BytesSchema) Type() int

Type returns a type constant for this BytesSchema.

func (*BytesSchema) Validate 

func (*BytesSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type CodeGenerator 

type CodeGenerator struct {
	// contains filtered or unexported fields
}

CodeGenerator is a code generation tool for structs from given Avro schemas.

func NewCodeGenerator 

func NewCodeGenerator(schemas []string) *CodeGenerator

NewCodeGenerator creates a new CodeGenerator for given Avro schemas.

func (*CodeGenerator) Generate 

func (codegen *CodeGenerator) Generate() (string, error)

Generate generates source code for Avro schemas specified on creation. The ouput is Go formatted source code that contains struct definitions for all given schemas. May return an error if code generation fails, e.g. due to unparsable schema.

type DataBlock 

type DataBlock struct {

	// Number of entries encoded in Data.
	NumEntries int64

	// Size of data buffer in bytes.
	BlockSize int

	// Number of unread entries in this DataBlock.
	BlockRemaining int64
	// contains filtered or unexported fields
}

DataBlock is a structure that holds a certain amount of entries and the actual buffer to read from.

type DataFileReader 

type DataFileReader struct {
	// contains filtered or unexported fields
}

DataFileReader is a reader for Avro Object Container Files. More here: https://avro.apache.org/docs/current/spec.html#Object+Container+Files

Example 
package main

import (
	"log"

	avro "github.com/kjuulh/go-avro"
)

type SomeStruct struct {
	Name string
}

func main() {
	// Create a reader open for reading on a data file.
	reader, err := avro.NewDataFileReader("filename.avro")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	for reader.HasNext() {
		var dest SomeStruct // or a *avro.GenericRecord
		if err := reader.Next(&dest); err != nil {
			// Error specific to decoding a single record
		}
		log.Printf("Decoded record %v", dest)
	}

	// If there was any error that stopped the reader loop, this is how we know
	if err := reader.Err(); err != nil {
		log.Fatal(err)
	}
}

Output:

func NewDataFileReader 

func NewDataFileReader(filename string, ignoreMe ...DatumReader) (*DataFileReader, error)

NewDataFileReader enables reading an object container file from the filesystem. May return an error if the file contains invalid data or is just missing.

The second DatumReader argument is deprecated, only there for source compatibility. Will be removed in an upcoming compatibility break.

func (*DataFileReader) Close 

func (reader *DataFileReader) Close() error

Close the underlying file if necessary.

Needed with filesystem files if you want to not leak filehandles. Returns any error in closing.

func (*DataFileReader) Err 

func (reader *DataFileReader) Err() error

Err returns the last encountered error.

Will not return io.EOF if that was the last error.

func (*DataFileReader) HasNext 

func (reader *DataFileReader) HasNext() bool

HasNext is used in a for loop to know you can continue on.

If there was an I/O or decoding error in decoding a block, then HasNext will be false, even if there might be more data in the data file.

It might be possible to recover from corrupted data by jumping to the next block by using the NextBlock() but this is not guaranteed.

func (*DataFileReader) Next 

func (reader *DataFileReader) Next(v interface{}) error

Next reads the next value from file and fills the given value with data.

v can be anything a DatumReader would accept, including a pointer to a struct that is compatible with this file's schema, an allocated *GenericRecord, or a **GenericRecord (library allocates for you)

Will error with io.EOF if you're past the end, loop HasNext() to prevent.

func (*DataFileReader) NextBlock 

func (reader *DataFileReader) NextBlock() error

NextBlock tells this DataFileReader to skip current block and move to next one.

This is not typically needed as the Next() loop will automatically advance to the next block for you.

May return an error if the block is malformed or io.EOF if no more blocks left to read.

type DataFileWriter 

type DataFileWriter struct {
	// contains filtered or unexported fields
}

DataFileWriter lets you write object container files.

func NewDataFileWriter 

func NewDataFileWriter(output io.Writer, schema Schema, datumWriter DatumWriter) (writer *DataFileWriter, err error)

NewDataFileWriter creates a new DataFileWriter for given output and schema using the given DatumWriter to write the data to that Writer. May return an error if writing fails.

func (*DataFileWriter) Close 

func (w *DataFileWriter) Close() error

Close this DataFileWriter. This is required to finish out the data file format. After Close() is called, this DataFileWriter cannot be used anymore.

func (*DataFileWriter) Flush 

func (w *DataFileWriter) Flush() error

Flush out any previously written datums to our underlying io.Writer. Does nothing if no datums had previously been written.

It's up to the library user to decide how often to flush; doing it often will spend a lot of time on tiny I/O but save memory.

func (*DataFileWriter) Write 

func (w *DataFileWriter) Write(v interface{}) error

Write out a single datum.

Encoded datums are buffered internally and will not be written to the underlying io.Writer until Flush() is called.

type DatumReader 

type DatumReader interface {
	// Reads a single structured entry using this DatumReader according to provided Schema.
	// Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type.
	// May return an error indicating a read failure.
	Read(v interface{}, dec Decoder) error
}

DatumReader is an interface that is responsible for reading structured data according to schema from a decoder

func NewDatumReader 

func NewDatumReader(schema Schema) DatumReader

NewDatumReader creates a DatumReader that can handle both GenericRecord and also aribtrary structs.

This is the preferred implementation at this point in time.

type DatumWriter 

type DatumWriter interface {
	// Write writes a single entry using this DatumWriter according to provided Schema.
	// Accepts a value to write and Encoder to write to.
	// May return an error indicating a write failure.
	Write(obj interface{}, enc Encoder) error
}

DatumWriter is an interface that is responsible for writing structured data according to schema to an encoder.

func NewDatumWriter 

func NewDatumWriter(schema Schema) DatumWriter

NewDatumWriter creates a DatumWriter that can handle both GenericRecord and also aribtrary structs.

This is the preferred implementation at this point in time.

type Decoder 

type Decoder interface {
	// Reads a null value. Returns a decoded value and an error if it occurs.
	ReadNull() (interface{}, error)

	// Reads a boolean value. Returns a decoded value and an error if it occurs.
	ReadBoolean() (bool, error)

	// Reads an in value. Returns a decoded value and an error if it occurs.
	ReadInt() (int32, error)

	// Reads a long value. Returns a decoded value and an error if it occurs.
	ReadLong() (int64, error)

	// Reads a float value. Returns a decoded value and an error if it occurs.
	ReadFloat() (float32, error)

	// Reads a double value. Returns a decoded value and an error if it occurs.
	ReadDouble() (float64, error)

	// Reads a bytes value. Returns a decoded value and an error if it occurs.
	ReadBytes() ([]byte, error)

	// Reads a string value. Returns a decoded value and an error if it occurs.
	ReadString() (string, error)

	// Reads an enum value (which is an Avro int value). Returns a decoded value and an error if it occurs.
	ReadEnum() (int32, error)

	// Reads and returns the size of the first block of an array. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call ArrayNext() to find out the number of items in the
	// next block. Returns a decoded value and an error if it occurs.
	ReadArrayStart() (int64, error)

	// Processes the next block of an array and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	ArrayNext() (int64, error)

	// Reads and returns the size of the first block of map entries. If call to this return non-zero, then the caller
	// should read the indicated number of items and then call MapNext() to find out the number of items in the
	// next block. Usage is similar to ReadArrayStart(). Returns a decoded value and an error if it occurs.
	ReadMapStart() (int64, error)

	// Processes the next block of map entries and returns the number of items in the block.
	// Returns a decoded value and an error if it occurs.
	MapNext() (int64, error)

	// Reads fixed sized binary object into the provided buffer.
	// Returns an error if it occurs.
	ReadFixed([]byte) error
}

Decoder is an interface that provides low-level support for deserializing Avro values.

func NewBinaryDecoder 

func NewBinaryDecoder(buf []byte) Decoder

NewBinaryDecoder creates a new BinaryDecoder to read from a given buffer.

func NewBinaryDecoderReader 

func NewBinaryDecoderReader(r io.Reader) Decoder

NewBinaryDecoderReader creates a new BinaryDecoder to read from a given io.Reader.

This decoder makes a lot of very small reads from the underlying io.Reader. If this is some high-latency object like a network socket or file, consider passing some sort of buffered reader like a bufio.Reader.

type DoubleSchema 

type DoubleSchema struct{}

DoubleSchema implements Schema and represents Avro double type.

func (*DoubleSchema) GetName 

func (*DoubleSchema) GetName() string

GetName returns a type name for this DoubleSchema.

func (*DoubleSchema) MarshalJSON 

func (*DoubleSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*DoubleSchema) Prop 

func (*DoubleSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for DoubleSchema.

func (*DoubleSchema) String 

func (*DoubleSchema) String() string

Returns a JSON representation of DoubleSchema.

func (*DoubleSchema) Type 

func (*DoubleSchema) Type() int

Type returns a type constant for this DoubleSchema.

func (*DoubleSchema) Validate 

func (*DoubleSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Encoder 

type Encoder interface {
	// Writes a null value. Doesn't actually do anything but may advance the state of Encoder implementation if it
	// is stateful.
	WriteNull(interface{})

	// Writes a boolean value.
	WriteBoolean(bool)

	// Writes an int value.
	WriteInt(int32)

	// Writes a long value.
	WriteLong(int64)

	// Writes a float value.
	WriteFloat(float32)

	// Writes a double value.
	WriteDouble(float64)

	// Writes a bytes value.
	WriteBytes([]byte)

	// Writes a string value.
	WriteString(string)

	// WriteArrayStart should be called when starting to serialize an array providing it with a number of items in
	// array block.
	WriteArrayStart(int64)

	// WriteArrayNext should be called after finishing writing an array block either passing it the number of items in
	// next block or 0 indicating the end of array.
	WriteArrayNext(int64)

	// WriteMapStart should be called when starting to serialize a map providing it with a number of items in
	// map block.
	WriteMapStart(int64)

	// WriteMapNext should be called after finishing writing a map block either passing it the number of items in
	// next block or 0 indicating the end of map.
	WriteMapNext(int64)

	// Writes raw bytes to this Encoder.
	WriteRaw([]byte)
}

Encoder is an interface that provides low-level support for serializing Avro values.

func NewBinaryEncoder 

func NewBinaryEncoder(buffer io.Writer) Encoder

NewBinaryEncoder creates a new BinaryEncoder that will write to a given io.Writer.

type EnumSchema 

type EnumSchema struct {
	Name       string
	Namespace  string
	Aliases    []string
	Doc        string
	Symbols    []string
	Properties map[string]interface{}
}

EnumSchema implements Schema and represents Avro enum type.

func (*EnumSchema) GetName 

func (s *EnumSchema) GetName() string

GetName returns an enum name for this EnumSchema.

func (*EnumSchema) MarshalJSON 

func (s *EnumSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*EnumSchema) Prop 

func (s *EnumSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*EnumSchema) String 

func (s *EnumSchema) String() string

String returns a JSON representation of EnumSchema.

func (*EnumSchema) Type 

func (*EnumSchema) Type() int

Type returns a type constant for this EnumSchema.

func (*EnumSchema) Validate 

func (*EnumSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FixedSchema 

type FixedSchema struct {
	Namespace  string
	Name       string
	Size       int
	Properties map[string]interface{}
}

FixedSchema implements Schema and represents Avro fixed type.

func (*FixedSchema) GetName 

func (s *FixedSchema) GetName() string

GetName returns a fixed name for this FixedSchema.

func (*FixedSchema) MarshalJSON 

func (s *FixedSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*FixedSchema) Prop 

func (s *FixedSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*FixedSchema) String 

func (s *FixedSchema) String() string

String returns a JSON representation of FixedSchema.

func (*FixedSchema) Type 

func (*FixedSchema) Type() int

Type returns a type constant for this FixedSchema.

func (*FixedSchema) Validate 

func (s *FixedSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type FloatSchema 

type FloatSchema struct{}

FloatSchema implements Schema and represents Avro float type.

func (*FloatSchema) GetName 

func (*FloatSchema) GetName() string

GetName returns a type name for this FloatSchema.

func (*FloatSchema) MarshalJSON 

func (*FloatSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*FloatSchema) Prop 

func (*FloatSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for FloatSchema.

func (*FloatSchema) String 

func (*FloatSchema) String() string

String returns a JSON representation of FloatSchema.

func (*FloatSchema) Type 

func (*FloatSchema) Type() int

Type returns a type constant for this FloatSchema.

func (*FloatSchema) Validate 

func (*FloatSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type GenericDatumReader 

type GenericDatumReader struct {
	// contains filtered or unexported fields
}

GenericDatumReader implements DatumReader and is used for filling GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) with data. Each value passed to Read is expected to be a pointer.

func NewGenericDatumReader 

func NewGenericDatumReader() *GenericDatumReader

NewGenericDatumReader creates a new GenericDatumReader.

func (*GenericDatumReader) Read 

func (reader *GenericDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single entry using this GenericDatumReader. Accepts a value to fill with data and a Decoder to read from. Given value MUST be of pointer type. May return an error indicating a read failure.

func (*GenericDatumReader) SetSchema 

func (reader *GenericDatumReader) SetSchema(schema Schema) DatumReader

SetSchema sets the schema for this GenericDatumReader to know the data structure. Note that it must be called before calling Read.

type GenericDatumWriter 

type GenericDatumWriter struct {
	// contains filtered or unexported fields
}

GenericDatumWriter implements DatumWriter and is used for writing GenericRecords or other Avro supported types (full list is: interface{}, bool, int32, int64, float32, float64, string, slices of any type, maps with string keys and any values, GenericEnums) to a given Encoder.

func NewGenericDatumWriter 

func NewGenericDatumWriter() *GenericDatumWriter

NewGenericDatumWriter creates a new GenericDatumWriter.

func (*GenericDatumWriter) SetSchema 

func (writer *GenericDatumWriter) SetSchema(schema Schema) DatumWriter

SetSchema sets the provided schema for this GenericDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*GenericDatumWriter) Write 

func (writer *GenericDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single entry using this GenericDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. May return an error indicating a write failure.

type GenericEnum 

type GenericEnum struct {
	// Avro enum symbols.
	Symbols []string
	// contains filtered or unexported fields
}

GenericEnum is a generic Avro enum representation. This is still subject to change and may be rethought.

func NewGenericEnum 

func NewGenericEnum(symbols []string) *GenericEnum

NewGenericEnum returns a new GenericEnum that uses provided enum symbols.

func (*GenericEnum) Get 

func (enum *GenericEnum) Get() string

Get gets the string value for this enum (e.g. symbol).

func (*GenericEnum) GetIndex 

func (enum *GenericEnum) GetIndex() int32

GetIndex gets the numeric value for this enum.

func (*GenericEnum) Set 

func (enum *GenericEnum) Set(symbol string)

Set sets the string value for this enum (e.g. symbol). Panics if the given symbol does not exist in this enum.

func (*GenericEnum) SetIndex 

func (enum *GenericEnum) SetIndex(index int32)

SetIndex sets the numeric value for this enum.

type GenericRecord 

type GenericRecord struct {
	// contains filtered or unexported fields
}

GenericRecord is a generic instance of a record schema. Fields are accessible by their name.

func NewGenericRecord 

func NewGenericRecord(schema Schema) *GenericRecord

NewGenericRecord creates a new GenericRecord.

func (*GenericRecord) Get 

func (gr *GenericRecord) Get(name string) interface{}

Get gets a value by its name.

func (*GenericRecord) Map 

func (gr *GenericRecord) Map() map[string]interface{}

Map returns a map representation of this GenericRecord.

func (*GenericRecord) Schema 

func (gr *GenericRecord) Schema() Schema

Schema returns a schema for this GenericRecord.

func (*GenericRecord) Set 

func (gr *GenericRecord) Set(name string, value interface{})

Set sets a value for a given name.

func (*GenericRecord) String 

func (gr *GenericRecord) String() string

String returns a JSON representation of this GenericRecord.

type IntSchema 

type IntSchema struct{}

IntSchema implements Schema and represents Avro int type.

func (*IntSchema) GetName 

func (*IntSchema) GetName() string

GetName returns a type name for this IntSchema.

func (*IntSchema) MarshalJSON 

func (*IntSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*IntSchema) Prop 

func (*IntSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for IntSchema.

func (*IntSchema) String 

func (*IntSchema) String() string

String returns a JSON representation of IntSchema.

func (*IntSchema) Type 

func (*IntSchema) Type() int

Type returns a type constant for this IntSchema.

func (*IntSchema) Validate 

func (*IntSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type LongSchema 

type LongSchema struct{}

LongSchema implements Schema and represents Avro long type.

func (*LongSchema) GetName 

func (*LongSchema) GetName() string

GetName returns a type name for this LongSchema.

func (*LongSchema) MarshalJSON 

func (*LongSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*LongSchema) Prop 

func (*LongSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for LongSchema.

func (*LongSchema) String 

func (*LongSchema) String() string

Returns a JSON representation of LongSchema.

func (*LongSchema) Type 

func (*LongSchema) Type() int

Type returns a type constant for this LongSchema.

func (*LongSchema) Validate 

func (*LongSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type MapSchema 

type MapSchema struct {
	Values     Schema
	Properties map[string]interface{}
}

MapSchema implements Schema and represents Avro map type.

func (*MapSchema) GetName 

func (*MapSchema) GetName() string

GetName returns a type name for this MapSchema.

func (*MapSchema) MarshalJSON 

func (s *MapSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*MapSchema) Prop 

func (s *MapSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*MapSchema) String 

func (s *MapSchema) String() string

String returns a JSON representation of MapSchema.

func (*MapSchema) Type 

func (*MapSchema) Type() int

Type returns a type constant for this MapSchema.

func (*MapSchema) Validate 

func (s *MapSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Marshaler 

type Marshaler interface {
	MarshalAvro(enc Encoder) error
}

Marshaler is an interface that may be implemented to avoid using runtime reflection during serialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

type NullSchema 

type NullSchema struct{}

NullSchema implements Schema and represents Avro null type.

func (*NullSchema) GetName 

func (*NullSchema) GetName() string

GetName returns a type name for this NullSchema.

func (*NullSchema) MarshalJSON 

func (*NullSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*NullSchema) Prop 

func (*NullSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for NullSchema.

func (*NullSchema) String 

func (*NullSchema) String() string

String returns a JSON representation of NullSchema.

func (*NullSchema) Type 

func (*NullSchema) Type() int

Type returns a type constant for this NullSchema.

func (*NullSchema) Validate 

func (*NullSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type RecordSchema 

type RecordSchema struct {
	Name       string   `json:"name,omitempty"`
	Namespace  string   `json:"namespace,omitempty"`
	Doc        string   `json:"doc,omitempty"`
	Aliases    []string `json:"aliases,omitempty"`
	Properties map[string]interface{}
	Fields     []*SchemaField `json:"fields"`
}

RecordSchema implements Schema and represents Avro record type.

func (*RecordSchema) GetName 

func (s *RecordSchema) GetName() string

GetName returns a record name for this RecordSchema.

func (*RecordSchema) MarshalJSON 

func (s *RecordSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*RecordSchema) Prop 

func (s *RecordSchema) Prop(key string) (interface{}, bool)

Prop gets a custom non-reserved property from this schema and a bool representing if it exists.

func (*RecordSchema) String 

func (s *RecordSchema) String() string

String returns a JSON representation of RecordSchema.

func (*RecordSchema) Type 

func (*RecordSchema) Type() int

Type returns a type constant for this RecordSchema.

func (*RecordSchema) Validate 

func (s *RecordSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type RecursiveSchema 

type RecursiveSchema struct {
	Actual *RecordSchema
}

RecursiveSchema implements Schema and represents Avro record type without a definition (e.g. that should be looked up).

func (*RecursiveSchema) GetName 

func (s *RecursiveSchema) GetName() string

GetName returns a record name for enclosed RecordSchema.

func (*RecursiveSchema) MarshalJSON 

func (s *RecursiveSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*RecursiveSchema) Prop 

func (*RecursiveSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for RecursiveSchema.

func (*RecursiveSchema) String 

func (s *RecursiveSchema) String() string

String returns a JSON representation of RecursiveSchema.

func (*RecursiveSchema) Type 

func (*RecursiveSchema) Type() int

Type returns a type constant for this RecursiveSchema.

func (*RecursiveSchema) Validate 

func (s *RecursiveSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Schema 

type Schema interface {
	// Returns an integer constant representing this schema type.
	Type() int

	// If this is a record, enum or fixed, returns its name, otherwise the name of the primitive type.
	GetName() string

	// Gets a custom non-reserved property from this schema and a bool representing if it exists.
	Prop(key string) (interface{}, bool)

	// Converts this schema to its JSON representation.
	String() string

	// Checks whether the given value is writeable to this schema.
	Validate(v reflect.Value) bool
}

Schema is an interface representing a single Avro schema (both primitive and complex).

func MustParseSchema 

func MustParseSchema(rawSchema string) Schema

MustParseSchema is like ParseSchema, but panics if the given schema cannot be parsed.

func ParseSchema 

func ParseSchema(rawSchema string) (Schema, error)

ParseSchema parses a given schema without provided schemas to reuse. Equivalent to call ParseSchemaWithResistry(rawSchema, make(map[string]Schema)) May return an error if schema is not parsable or has insufficient information about any type.

func ParseSchemaFile 

func ParseSchemaFile(file string) (Schema, error)

ParseSchemaFile parses a given file. May return an error if schema is not parsable or file does not exist.

func ParseSchemaWithRegistry 

func ParseSchemaWithRegistry(rawSchema string, schemas map[string]Schema) (Schema, error)

ParseSchemaWithRegistry parses a given schema using the provided registry for type lookup. Registry will be filled up during parsing. May return an error if schema is not parsable or has insufficient information about any type.

func Prepare 

func Prepare(schema Schema) Schema

Prepare optimizes a schema for decoding/encoding.

It makes a recursive copy of the schema given and returns an immutable wrapper of the schema with some optimizations applied.

type SchemaField 

type SchemaField struct {
	Name       string      `json:"name,omitempty"`
	Doc        string      `json:"doc,omitempty"`
	Default    interface{} `json:"default"`
	Type       Schema      `json:"type,omitempty"`
	Properties map[string]interface{}
}

SchemaField represents a schema field for Avro record.

func (*SchemaField) MarshalJSON 

func (s *SchemaField) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema field as JSON.

func (*SchemaField) Prop 

func (this *SchemaField) Prop(key string) (interface{}, bool)

Gets a custom non-reserved property from this schemafield and a bool representing if it exists.

func (*SchemaField) String 

func (s *SchemaField) String() string

String returns a JSON representation of SchemaField.

type SpecificDatumReader 

type SpecificDatumReader struct {
	// contains filtered or unexported fields
}

SpecificDatumReader implements DatumReader and is used for filling Go structs with data. Each value passed to Read is expected to be a pointer.

func NewSpecificDatumReader 

func NewSpecificDatumReader() *SpecificDatumReader

NewSpecificDatumReader creates a new SpecificDatumReader.

func (*SpecificDatumReader) Read 

func (reader *SpecificDatumReader) Read(v interface{}, dec Decoder) error

Read reads a single structured entry using this SpecificDatumReader. Accepts a Go struct with exported fields to fill with data and a Decoder to read from. Given value MUST be of pointer type. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a read failure.

func (*SpecificDatumReader) SetSchema 

func (reader *SpecificDatumReader) SetSchema(schema Schema) DatumReader

SetSchema sets the schema for this SpecificDatumReader to know the data structure. Note that it must be called before calling Read.

type SpecificDatumWriter 

type SpecificDatumWriter struct {
	// contains filtered or unexported fields
}

SpecificDatumWriter implements DatumWriter and is used for writing Go structs in Avro format.

Example (Basic) 
package main

import (
	"bytes"
	"log"

	avro "github.com/kjuulh/go-avro"
)

var someSchema avro.Schema

type SomeStruct struct {
	Name string
}

func main() {
	writer := avro.NewSpecificDatumWriter()
	writer.SetSchema(someSchema)

	var buf bytes.Buffer
	v := &SomeStruct{Name: "abc"}

	if err := writer.Write(v, avro.NewBinaryEncoder(&buf)); err != nil {
		log.Fatal(err) // i/o errors OR encoding errors
	}

	// buf.Bytes() now contains the encoded value of 'v'
}

Output:
Example (Full) 
package main

import (
	"bytes"
	"fmt"
	"log"

	avro "github.com/kjuulh/go-avro"
)

func main() {
	// Parse a schema from JSON to get the schema object.
	schema, err := avro.ParseSchema(`{
		"type": "record",
		"name": "Person",
		"fields": [
			{"name": "first_name", "type": "string"},
			{"name": "age", "type": "int"}
		]
	}`)
	if err != nil {
		log.Fatal(err)
	}

	// This is a struct which supports the above schema.
	type Person struct {
		FirstName string `avro:"first_name"` // the avro: tag specifies which avro field matches this field.
		Age       int32  `avro:"age"`
	}

	// Create a SpecificDatumWriter, which you can re-use multiple times.
	writer := avro.NewSpecificDatumWriter()
	writer.SetSchema(schema)

	// Write a person to a byte buffer as avro.
	person := &Person{
		FirstName: "Bob",
		Age:       48,
	}

	var buf bytes.Buffer
	encoder := avro.NewBinaryEncoder(&buf)
	err = writer.Write(person, encoder)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(buf.Bytes())
}

Output:

[6 66 111 98 96]

func NewSpecificDatumWriter 

func NewSpecificDatumWriter() *SpecificDatumWriter

NewSpecificDatumWriter creates a new SpecificDatumWriter.

func (*SpecificDatumWriter) SetSchema 

func (writer *SpecificDatumWriter) SetSchema(schema Schema) DatumWriter

SetSchema sets the provided schema for this SpecificDatumWriter to know the data structure. Note that it must be called before calling Write.

func (*SpecificDatumWriter) Write 

func (writer *SpecificDatumWriter) Write(obj interface{}, enc Encoder) error

Write writes a single Go struct using this SpecificDatumWriter according to provided Schema. Accepts a value to write and Encoder to write to. Field names should match field names in Avro schema but be exported (e.g. "some_value" in Avro schema is expected to be Some_value in struct) or you may provide Go struct tags to explicitly show how to map fields (e.g. if you want to map "some_value" field of type int to SomeValue in Go struct you should define your struct field as follows: SomeValue int32 `avro:"some_field"`). May return an error indicating a write failure.

type StringSchema 

type StringSchema struct{}

StringSchema implements Schema and represents Avro string type.

func (*StringSchema) GetName 

func (*StringSchema) GetName() string

GetName returns a type name for this StringSchema.

func (*StringSchema) MarshalJSON 

func (*StringSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON. Never returns an error.

func (*StringSchema) Prop 

func (*StringSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for StringSchema.

func (*StringSchema) String 

func (*StringSchema) String() string

Returns a JSON representation of StringSchema.

func (*StringSchema) Type 

func (*StringSchema) Type() int

Type returns a type constant for this StringSchema.

func (*StringSchema) Validate 

func (*StringSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type UnionSchema 

type UnionSchema struct {
	Types []Schema
}

UnionSchema implements Schema and represents Avro union type.

func (*UnionSchema) GetName 

func (*UnionSchema) GetName() string

GetName returns a type name for this UnionSchema.

func (*UnionSchema) GetType 

func (s *UnionSchema) GetType(v reflect.Value) int

GetType gets the index of actual union type for a given value.

func (*UnionSchema) MarshalJSON 

func (s *UnionSchema) MarshalJSON() ([]byte, error)

MarshalJSON serializes the given schema as JSON.

func (*UnionSchema) Prop 

func (*UnionSchema) Prop(key string) (interface{}, bool)

Prop doesn't return anything valuable for UnionSchema.

func (*UnionSchema) String 

func (s *UnionSchema) String() string

String returns a JSON representation of UnionSchema.

func (*UnionSchema) Type 

func (*UnionSchema) Type() int

Type returns a type constant for this UnionSchema.

func (*UnionSchema) Validate 

func (s *UnionSchema) Validate(v reflect.Value) bool

Validate checks whether the given value is writeable to this schema.

type Unmarshaler 

type Unmarshaler interface {
	UnmarshalAvro(dec Decoder) error
}

Unmarshaler is an interface that may be implemented to avoid using runtime reflection during deserialization. Implementing it is optional and may be used as an optimization. Falls back to using reflection if not implemented.

Source Files

View all Source files

Directories

Path Synopsis
examples
data_file
generic_datum
load_schema
specific_datum
genericreader

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.