Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Extensions ¶
type Extensions []string
Extensions is used to tokenize snippets in directories using the list of file extensions.
func (Extensions) ReadLines ¶ added in v0.0.9
func (e Extensions) ReadLines(dirs ...string) apoco.StreamFunc
ReadLines returns a stream function that reads snippet files (identyfied by the given file extensions) and returns a stream of line tokens.
If a extension ends with `.txt`, one line is read from the text file (no confidences); if the file ends with `.json`, calamari's extended data format is assumed. Otherwise the file is read as a TSV file expecting one char and its confidence on each line.
func (Extensions) Tokenize ¶
func (e Extensions) Tokenize(dirs ...string) apoco.StreamFunc
Tokenize tokenizes tokens from line snippets (identyfied by the given file extensions) and alignes them accordingly. It is a shorthand for piping ReadLines into TokenizeLines.
func (Extensions) TokenizeLines ¶ added in v0.0.9
func (e Extensions) TokenizeLines() apoco.StreamFunc
TokenizeLines returns a stream function that tokenizes and aligns line tokens.
Source Files