textio

Documentation

Overview

Package textio contains transforms for reading and writing text files.

Index

• func Immediate(s beam.Scope, filename string) (beam.PCollection, error)
• func Read(s beam.Scope, glob string) beam.PCollection
• func ReadAll(s beam.Scope, col beam.PCollection) beam.PCollection
• func ReadAllSdf(s beam.Scope, col beam.PCollection) beam.PCollection
• func ReadSdf(s beam.Scope, glob string) beam.PCollection
• func Write(s beam.Scope, filename string, col beam.PCollection)

Functions

func Immediate

func Immediate(s beam.Scope, filename string) (beam.PCollection, error)

Immediate reads a local file at pipeline construction-time and embeds the data into a I/O-free pipeline source. Should be used for small files only.

func Read

func Read(s beam.Scope, glob string) beam.PCollection

Read reads a set of file and returns the lines as a PCollection<string>. The newlines are not part of the lines.

func ReadAll

func ReadAll(s beam.Scope, col beam.PCollection) beam.PCollection

ReadAll expands and reads the filename given as globs by the incoming PCollection<string>. It returns the lines of all files as a single PCollection<string>. The newlines are not part of the lines.

func ReadAllSdf

func ReadAllSdf(s beam.Scope, col beam.PCollection) beam.PCollection

ReadAllSdf is a variation of ReadAll implemented via SplittableDoFn. This should result in increased performance with runners that support splitting.

func ReadSdf

func ReadSdf(s beam.Scope, glob string) beam.PCollection

ReadSdf is a variation of Read implemented via SplittableDoFn. This should result in increased performance with runners that support splitting.

func Write

func Write(s beam.Scope, filename string, col beam.PCollection)

Write writes a PCollection<string> to a file as separate lines. The writer add a newline after each element.