documentio

Documentation

Overview

tfidf provides IO functions to read text files and also remembers the source of the text.

Index

• func Read(ctx context.Context, s beam.Scope, input string, corpusLen int, filter string, ...) beam.PCollection
• type CorpusEntry
• type DocEntry

Functions

func Read

func Read(ctx context.Context, s beam.Scope, input string, corpusLen int, filter string, entries beam.PCollection) beam.PCollection

Read a PCollection of CorpusEntry and return a PCollection of DocEntry with the contents of the file contained in the Text field.

Types

type CorpusEntry

type CorpusEntry struct {
	RawFile    string `beam:"rawFile"`
	DocumentId string `beam:"glossFile"`
	ColFile    string `beam:"colFile"`
}

CorpusEntry contains metadata for a document that text will be read from

type DocEntry

type DocEntry struct {
	Text       string `beam:"text"`
	DocumentId string `beam:"glossFile"`
	ColFile    string `beam:"colFile"`
	CorpusLen  int    `beam:"corpusLen"`
}

DocEntry contains all text and metadata for the document that it was extracted from