avro

Documentation

Overview

Package avro reads Avro OCF files and presents the extracted data as records

Index

• Variables
• func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error)
• type OCFReader
  • func NewOCFReader(r io.Reader, opts ...Option) (*OCFReader, error)
  • func (r *OCFReader) AvroSchema() string
  • func (r *OCFReader) Close()
  • func (r *OCFReader) Err() error
  • func (r *OCFReader) Metrics() string
  • func (r *OCFReader) Next() bool
  • func (r *OCFReader) OCFRecordsReadCount() int64
  • func (r *OCFReader) Record() arrow.Record
  • func (r *OCFReader) Release()
  • func (r *OCFReader) Retain()
  • func (rr *OCFReader) Reuse(r io.Reader, opts ...Option) error
  • func (r *OCFReader) Schema() *arrow.Schema
• type Option
  • func WithAllocator(mem memory.Allocator) Option
  • func WithChunk(n int) Option
  • func WithReadCacheSize(n int) Option
  • func WithRecordCacheSize(n int) Option
  • func WithSchemaEdit(method, path string, value any) Option

Variables

var ErrMismatchFields = errors.New("arrow/avro: number of records mismatch")

var (
	ErrNullStructData = errors.New("null struct data")
)

Functions

func ArrowSchemaFromAvro

func ArrowSchemaFromAvro(schema avro.Schema) (s *arrow.Schema, err error)

ArrowSchemaFromAvro returns a new Arrow schema from an Avro schema

Types

type OCFReader

type OCFReader struct {
	// contains filtered or unexported fields
}

Reader wraps goavro/OCFReader and creates array.Records from a schema.

func NewOCFReader

func NewOCFReader(r io.Reader, opts ...Option) (*OCFReader, error)

NewReader returns a reader that reads from an Avro OCF file and creates arrow.Records from the converted avro data.

func (*OCFReader) AvroSchema

func (r *OCFReader) AvroSchema() string

AvroSchema returns the Avro schema of the Avro OCF

func (*OCFReader) Close

func (r *OCFReader) Close()

Close closes the OCFReader's Avro record read cache and converted Arrow record cache. OCFReader must be closed if the Avro OCF's records have not been read to completion.

func (*OCFReader) Err

func (r *OCFReader) Err() error

Err returns the last error encountered during the iteration over the underlying Avro file.

func (*OCFReader) Metrics

func (r *OCFReader) Metrics() string

Metrics returns the maximum queue depth of the Avro record read cache and of the converted Arrow record cache.

func (*OCFReader) Next

func (r *OCFReader) Next() bool

Next returns whether a Record can be received from the converted record queue. The user should check Err() after call to Next that return false to check if an error took place.

func (*OCFReader) OCFRecordsReadCount

func (r *OCFReader) OCFRecordsReadCount() int64

OCFRecordsReadCount returns the number of Avro datum that were read from the Avro file.

func (*OCFReader) Record

func (r *OCFReader) Record() arrow.Record

Record returns the current record that has been extracted from the underlying Avro OCF file. It is valid until the next call to Next.

func (*OCFReader) Release

func (r *OCFReader) Release()

Release decreases the reference count by 1. When the reference count goes to zero, the memory is freed. Release may be called simultaneously from multiple goroutines.

func (*OCFReader) Retain

func (r *OCFReader) Retain()

Retain increases the reference count by 1. Retain may be called simultaneously from multiple goroutines.

func (*OCFReader) Reuse

func (rr *OCFReader) Reuse(r io.Reader, opts ...Option) error

Reuse allows the OCFReader to be reused to read another Avro file provided the new Avro file has an identical schema.

func (*OCFReader) Schema

func (r *OCFReader) Schema() *arrow.Schema

Schema returns the converted Arrow schema of the Avro OCF

type Option

type Option func(config)

Option configures an Avro reader/writer.

func WithAllocator

func WithAllocator(mem memory.Allocator) Option

WithAllocator specifies the Arrow memory allocator used while building records.

func WithChunk

func WithChunk(n int) Option

WithChunk specifies the chunk size used while reading Avro OCF files.

If n is zero or 1, no chunking will take place and the reader will create one record per row. If n is greater than 1, chunks of n rows will be read. If n is negative, the reader will load the whole Avro OCF file into memory and create one big record with all the rows.

func WithReadCacheSize

func WithReadCacheSize(n int) Option

WithReadCacheSize specifies the size of the OCF record decode queue, default value is 500.

func WithRecordCacheSize

func WithRecordCacheSize(n int) Option

WithRecordCacheSize specifies the size of the converted Arrow record queue, default value is 1.

func WithSchemaEdit

func WithSchemaEdit(method, path string, value any) Option

WithSchemaEdit specifies modifications to the Avro schema. Supported methods are 'set' and 'delete'. Set sets the value for the specified path. Delete deletes the value for the specified path. A path is in dot syntax, such as "fields.1" or "fields.0.type". The modified Avro schema is validated before conversion to Arrow schema - NewOCFReader will return an error if the modified schema cannot be parsed.