goavro

README

goavro

Documentation

Index

• Constants
• Variables
• func Union(name string, datum interface{}) interface{}
• type BinaryCoder
• type BinaryDecoder
• type BinaryEncoder
• type Codec
  • func NewCodec(schemaSpecification string) (*Codec, error)
  • func (c Codec) BinaryDecode(buf []byte) (interface{}, []byte, error)
  • func (c Codec) BinaryEncode(buf []byte, datum interface{}) ([]byte, error)
  • func (c Codec) TextDecode(buf []byte) (interface{}, []byte, error)
  • func (c Codec) TextEncode(buf []byte, datum interface{}) ([]byte, error)
• type Compression
• type ErrInvalidName
  • func (e ErrInvalidName) Error() string
• type OCFReader
  • func NewOCFReader(ior io.Reader) (*OCFReader, error)
  • func (ocfr *OCFReader) Codec() *Codec
  • func (ocfr *OCFReader) CompressionID() Compression
  • func (ocfr *OCFReader) CompressionName() string
  • func (ocfr *OCFReader) Err() error
  • func (ocfr *OCFReader) Read() (interface{}, error)
  • func (ocfr *OCFReader) Scan() bool
  • func (ocfr *OCFReader) Schema() string
• type OCFWriter
  • func NewOCFWriter(config OCFWriterConfig) (*OCFWriter, error)
  • func (ocf *OCFWriter) Append(data []interface{}) error
• type OCFWriterConfig
• type TextCoder
• type TextDecoder
• type TextEncoder

Constants

const (
	// CompressionNullLabel is used when OCF blocks are not compressed.
	CompressionNullLabel = "null"

	// CompressionDeflateLabel is used when OCF blocks are compressed using the
	// deflate algorithm.
	CompressionDeflateLabel = "deflate"

	// CompressionSnappyLabel is used when OCF blocks are compressed using the
	// snappy algorithm.
	CompressionSnappyLabel = "snappy"
)

Variables

var (
	// MaxBlockCount is the maximum number of data items allowed in a single
	// binary block that will be decoded from a binary stream. This check is to
	// ensure decoding binary data will not cause the library to over allocate
	// RAM, potentially creating a denial of service on the system.
	//
	// If a particular application needs to decode binary Avro data that
	// potentially has more data items in a single block, then this variable may
	// be modified at your discretion.
	MaxBlockCount = int64(math.MaxInt32)

	// MaxBlockSize is the maximum number of bytes that will be allocated for a
	// single block of data items when decoding from a binary stream. This check
	// is to ensure decoding binary data will not cause the library to over
	// allocate RAM, potentially creating a denial of service on the system.
	//
	// If a particular application needs to decode binary Avro data that
	// potentially has more bytes in a single block, then this variable may be
	// modified at your discretion.
	MaxBlockSize = int64(math.MaxInt32)
)

Functions

func Union

func Union(name string, datum interface{}) interface{}

Union wraps a datum value in a map for encoding as a Union, as required by Union encoder.

Types

type BinaryCoder

type BinaryCoder interface {
	BinaryDecoder
	BinaryEncoder
}

BinaryCoder interface describes types that expose both the BinaryDecode and the BinaryEncode methods.

type BinaryDecoder

type BinaryDecoder interface {
	BinaryDecode([]byte) (interface{}, []byte, error)
}

BinaryDecoder interface describes types that expose the BinaryDecode method.

type BinaryEncoder

type BinaryEncoder interface {
	BinaryEncode([]byte, interface{}) ([]byte, error)
}

BinaryEncoder interface describes types that expose the BinaryEncode method.

type Codec

type Codec struct {
	// contains filtered or unexported fields
}

Codec supports decoding binary and text Avro data to Go native data types, and conversely encoding Go native data types to binary or text Avro data. A Codec is created as a stateless structure that can be safely used in multiple go routines simultaneously.

func NewCodec

func NewCodec(schemaSpecification string) (*Codec, error)

NewCodec returns a Codec used to decode binary and text Avro data to Go native data types, and conversely encode Go native types to binary and text Avro data. The returned Codec is a stateless structure that can be safely used in multiple go routines simultaneously. The returned Codec will only be able to decode and encode data that adheres to the supplied schema specification string.

func (Codec) BinaryDecode

func (c Codec) BinaryDecode(buf []byte) (interface{}, []byte, error)

BinaryDecode converts Avro data in binary format from the provided byte slice to Go native data types in accordance with the Avro schema supplied when creating the Codec. On success, it returns the decoded datum, along with a new byte slice with the decoded bytes consumed, and a nil error value. On error, it returns nil for the datum value, the original byte slice, and the error message.

func (Codec) BinaryEncode

func (c Codec) BinaryEncode(buf []byte, datum interface{}) ([]byte, error)

BinaryEncode converts Go native data types to Avro data in binary format in accordance with the Avro schema supplied when creating the Codec. It is supplied a byte slice to which to append the encoded data and the actual data to encode. On success, it returns a new byte slice with the encoded bytes appended, and a nil error value. On error, it returns the original byte slice, and the error message.

func (Codec) TextDecode

func (c Codec) TextDecode(buf []byte) (interface{}, []byte, error)

TextDecode converts Avro data in JSON text format from the provided byte slice to Go native data types in accordance with the Avro schema supplied when creating the Codec. On success, it returns the decoded datum, along with a new byte slice with the decoded bytes consumed, and a nil error value. On error, it returns nil for the datum value, the original byte slice, and the error message.

func (Codec) TextEncode

func (c Codec) TextEncode(buf []byte, datum interface{}) ([]byte, error)

TextEncode converts Go native data types to Avro data in JSON text format in accordance with the Avro schema supplied when creating the Codec. It is supplied a byte slice to which to append the encoded data and the actual data to encode. On success, it returns a new byte slice with the encoded bytes appended, and a nil error value. On error, it returns the original byte slice, and the error message.

type Compression

type Compression uint8

Compression are values used to specify compression algorithm used to compress and decompress Avro Object Container File (OCF) streams.

const (
	// CompressionNull is used when OCF blocks are not compressed.
	CompressionNull Compression = iota

	// CompressionDeflate is used when OCF blocks are compressed using the
	// deflate algorithm.
	CompressionDeflate

	// CompressionSnappy is used when OCF blocks are compressed using the snappy
	// algorithm.
	CompressionSnappy
)

type ErrInvalidName

type ErrInvalidName struct {
	Message string
}

ErrInvalidName is the error returned when one or more parts of an Avro name is invalid.

func (ErrInvalidName) Error

func (e ErrInvalidName) Error() string

type OCFReader

type OCFReader struct {
	// contains filtered or unexported fields
}

OCFReader structure is used to read Object Container Files (OCF).

func NewOCFReader

func NewOCFReader(ior io.Reader) (*OCFReader, error)

NewOCFReader initializes and returns a new structure used to read an Avro Object Container File (OCF).

func example(ior io.Reader) error {
	ocfr, err := goavro.NewOCFReader(ior)
	if err != nil {
		return err
	}
	for ocfr.Scan() {
		datum, err := ocfr.Read()
		if err != nil {
			return err
		}
		fmt.Println(datum)
	}
	return ocfr.Err()
}

func (*OCFReader) Codec

func (ocfr *OCFReader) Codec() *Codec

Codec returns the codec found within the OCF file.

func (*OCFReader) CompressionID

func (ocfr *OCFReader) CompressionID() Compression

CompressionID returns the ID of the compression algorithm found within the OCF file.

func (*OCFReader) CompressionName

func (ocfr *OCFReader) CompressionName() string

CompressionName returns the name of the compression algorithm found within the OCF file.

func (*OCFReader) Err

func (ocfr *OCFReader) Err() error

Err returns the last error encountered while reading the OCF file. It does not reset the read error.

func (*OCFReader) Read

func (ocfr *OCFReader) Read() (interface{}, error)

Read consumes one data item from the Avro OCF stream and returns it. Read is designed to be called only once after each invocation of the Scan method. See the documentation for goavro.NewOCFReader for an example of how to use Read.

func (*OCFReader) Scan

func (ocfr *OCFReader) Scan() bool

Scan returns true when there is at least one more data item to be read from the Avro OCF. Scan ought to be called prior to calling the Read method each time the Read method is invoked. See the documentation for goavro.NewOCFReader for an example of how to use Scan.

func (*OCFReader) Schema

func (ocfr *OCFReader) Schema() string

Schema returns the schema found within the OCF file.

type OCFWriter

type OCFWriter struct {
	// contains filtered or unexported fields
}

OCFWriter is used to create an Avro Object Container File (OCF).

func NewOCFWriter

func NewOCFWriter(config OCFWriterConfig) (*OCFWriter, error)

NewOCFWriter returns a newly created OCFWriter which may be used to create an Avro Object Container File (OCF).

func (*OCFWriter) Append

func (ocf *OCFWriter) Append(data []interface{}) error

Append appends one or more data items to an OCF file in a block. If there are more data items in the slice than MaxBlockCount allows, the data slice will be chunked into multiple blocks, each not having more than MaxBlockCount items.

type OCFWriterConfig

type OCFWriterConfig struct {
	// W specifies the io.Writer to send the encode the data, (required).
	W io.Writer

	// Schema specifies the Avro schema for the data to be encoded, (required).
	Schema string

	// Codec specifies the compression codec used, (optional). If omitted,
	// defaults to "null" codec.
	Compression Compression
}

OCFWriterConfig is used to specify creation parameters for OCFWriter.

type TextCoder

type TextCoder interface {
	TextDecoder
	TextEncoder
}

TextCoder interface describes types that expose both the TextDecode and the TextEncode methods.

type TextDecoder

type TextDecoder interface {
	TextDecode([]byte) (interface{}, []byte, error)
}

TextDecoder interface describes types that expose the TextDecode method.

type TextEncoder

type TextEncoder interface {
	TextEncode([]byte, interface{}) ([]byte, error)
}

TextEncoder interface describes types that expose the TextEncode method.