beamio

Documentation

Overview

Package beamio provides Beam transformations for common IO patterns.

Index

• func ComputeShards(s beam.Scope, kv beam.PCollection, opts stats.Opts) beam.PCollection
• func EncodeKeyValues(s beam.Scope, tables ...beam.PCollection) beam.PCollection
• func ReadEntries(ctx context.Context, s beam.Scope, fileOrDir string) (beam.PCollection, error)
• func WriteLevelDB(s beam.Scope, path string, opts stats.Opts, tables ...beam.PCollection)
• type KeyValue

Functions

func ComputeShards

func ComputeShards(s beam.Scope, kv beam.PCollection, opts stats.Opts) beam.PCollection

ComputeShards assigns shards to KeyValues. Input is a PCollection of beamio.KeyValue, output is (int, beamio.KeyValue).

func EncodeKeyValues

func EncodeKeyValues(s beam.Scope, tables ...beam.PCollection) beam.PCollection

EncodeKeyValues encodes each PCollection of KVs into encoded KeyValues and flattens all entries into a single PCollection.

func ReadEntries

func ReadEntries(ctx context.Context, s beam.Scope, fileOrDir string) (beam.PCollection, error)

ReadEntries reads a set of *spb.Entry messages into a PCollection from the given file, or files stored in a directory. The file can be part of any filesystem registered with the beam/io/filesystem package and can either be a delimited protobuf stream or a Riegeli file.

func WriteLevelDB

func WriteLevelDB(s beam.Scope, path string, opts stats.Opts, tables ...beam.PCollection)

WriteLevelDB writes a set of PCollections containing KVs to a new LevelDB at the given path. Each KV is serialized and stored as a single LevelDB key-value entry according to their enclosing PCollection's beam.Coder. Each table may have different KV types. Keys must be unique across all PCollections.

Types

type KeyValue

type KeyValue struct {
	Key   []byte `json:"k"`
	Value []byte `json:"v"`
}

A KeyValue is a concrete form of a Beam KV.