avroio

Documentation

Overview

Package avroio contains transforms for reading and writing avro files.

Index

• func Read(s beam.Scope, glob string, t reflect.Type) beam.PCollection
• func Write(s beam.Scope, filename, schema string, col beam.PCollection)

Functions

func Read

func Read(s beam.Scope, glob string, t reflect.Type) beam.PCollection

Read reads a set of files and returns lines as a PCollection<elem> based on the internal avro schema of the file. A type - reflect.TypeOf( YourType{} ) - with JSON tags can be defined or if you wish to return the raw JSON string, use - reflect.TypeOf("") -

func Write

func Write(s beam.Scope, filename, schema string, col beam.PCollection)

Write writes a PCollection<string> to an AVRO file. Write expects a JSON string with a matching AVRO schema. the process will fail if the schema does not match the JSON provided