container

Documentation

Overview

Container provides a Writer which is capable of serializing gogen-avro structs and writing them in the Avro Object Container File (OCF) format

Index

• type AvroRecord
• type CloseableResettableWriter
• type Codec
• type Reader
  • func NewReader(r io.Reader) (*Reader, error)
  • func (r *Reader) AvroContainerSchema() []byte
  • func (r *Reader) Read(b []byte) (n int, err error)
• type Writer
  • func NewWriter(writer io.Writer, codec Codec, recordsPerBlock int64, schema string) (*Writer, error)
  • func (avroWriter *Writer) Flush() error
  • func (avroWriter *Writer) WriteRecord(record AvroRecord) error

Types

type AvroRecord

type AvroRecord interface {
	Serialize(io.Writer) error
	Schema() string
}

AvroRecord is an interface fulfilled by the structs generated by gogen-avro for Avro records.

type CloseableResettableWriter

type CloseableResettableWriter interface {
	Close() error
	Reset(io.Writer)
}

type Codec

type Codec string

A Codec specifies how the blocks within a container file should be compressed.

const (
	// No compression
	Null Codec = "null"
	// Deflate compression
	Deflate Codec = "deflate"
	// Snappy compression
	Snappy Codec = "snappy"
)

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

*

• Reader provides an experimental Object Container File reader to load Avro container files. You can
• give this an OCF and it'll transparently decode a block at a time, so you can pass this to your
• generated deserializers. See `test/primitive/container_test.go` for an example. *
• Note: This is experimental and the interface may change or be deprecated. Right now gogen-avro
• only supports deserializing with the exact schema you used to serialize - there is no
• support for adding/removing fields or changing types.

func NewReader

func NewReader(r io.Reader) (*Reader, error)

func (*Reader) AvroContainerSchema

func (r *Reader) AvroContainerSchema() []byte

func (*Reader) Read

func (r *Reader) Read(b []byte) (n int, err error)

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer wraps an io.Writer and writes the file and block-level framing required for an OCF file

func NewWriter

func NewWriter(writer io.Writer, codec Codec, recordsPerBlock int64, schema string) (*Writer, error)

Create a new Writer wrapping the provided io.Writer with the given Codec and number of records per block. The Writer will lazily write the container file header when WriteRecord is called the first time. You must call Flush on the Writer before closing the underlying io.Writer, to ensure the final block is written. A schema string must be passed to ensure that a correct header is written even if no records are written. This is required to produce valid empty Avro container files.

func (*Writer) Flush

func (avroWriter *Writer) Flush() error

Write the current block to the file if it has been filled. It is best-practise to always call this before the underlying io.Writer is closed.

func (*Writer) WriteRecord

func (avroWriter *Writer) WriteRecord(record AvroRecord) error

Write an AvroRecord to the container file. All gogen-avro generated structs fulfill the AvroRecord interface. Note that all records in a given container file must be of the same Avro type.