container

Documentation

Overview

Container provides a Reader and Writer which serialize and deserialize gogen-avro structs to the Avro Object Container File (OCF) format. These are low-level primitives you probably don't need to use directly.

Index

• Variables
• type AvroRecord
• type CloseableResettableWriter
• type Codec
• type Reader
  • func NewReader(r io.Reader) (*Reader, error)
  • func (r *Reader) AvroContainerSchema() []byte
  • func (r *Reader) Read(b []byte) (n int, err error)
• type Writer
  • func NewWriter(writer io.Writer, codec Codec, recordsPerBlock int64, schema string) (*Writer, error)
  • func (avroWriter *Writer) BlockBufferSize() int
  • func (avroWriter *Writer) Flush() error
  • func (avroWriter *Writer) WriteRecord(record AvroRecord) error

Variables

var (
	LoggingEnabled = false
)

Types

type AvroRecord

type AvroRecord interface {
	Serialize(io.Writer) error
	Schema() string
}

AvroRecord is an interface fulfilled by the structs generated by gogen-avro for Avro records.

type CloseableResettableWriter

type CloseableResettableWriter interface {
	Close() error
	Reset(io.Writer)
}

type Codec

type Codec string

A Codec specifies how the blocks within a container file should be compressed.

const (
	// No compression
	Null Codec = "null"

	// Deflate compression
	Deflate Codec = "deflate"

	// Snappy compression
	Snappy Codec = "snappy"
)

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is a low-level primitive for reading the OCF framing of a file. Generally you can create a Reader using the `New<RecordType>Reader` method generate for every record type.

func NewReader

func NewReader(r io.Reader) (*Reader, error)

func (*Reader) AvroContainerSchema

func (r *Reader) AvroContainerSchema() []byte

func (*Reader) Read

func (r *Reader) Read(b []byte) (n int, err error)

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer wraps an io.Writer and writes the file and block-level framing required for an OCF file. You can create a Writer for a given struct by calling the generated method `New<RecordType>Writer`.

func NewWriter

func NewWriter(writer io.Writer, codec Codec, recordsPerBlock int64, schema string) (*Writer, error)

Create a new Writer wrapping the provided io.Writer with the given Codec and number of records per block. The Writer will lazily write the container file header when WriteRecord is called the first time. You must call Flush on the Writer before closing the underlying io.Writer, to ensure the final block is written. A schema string must be passed to ensure that a correct header is written even if no records are written. This is required to produce valid empty Avro container files.

func (*Writer) BlockBufferSize

func (avroWriter *Writer) BlockBufferSize() int

Get the current block buffer size. caller might trigger an early Flush if current block size gets huge.

func (*Writer) Flush

func (avroWriter *Writer) Flush() error

Write the current block to the file if it has been filled. It is best-practise to always call this before the underlying io.Writer is closed.

func (*Writer) WriteRecord

func (avroWriter *Writer) WriteRecord(record AvroRecord) error

Write an AvroRecord to the container file. All gogen-avro generated structs fulfill the AvroRecord interface. Note that all records in a given container file must be of the same Avro type.