snippets

Documentation

Index

• type Extensions
  • func (e Extensions) ReadLines(dirs ...string) apoco.StreamFunc
  • func (e Extensions) TokenizeLines(ctx context.Context, in <-chan apoco.Token, out chan<- apoco.Token) error

Types

type Extensions

type Extensions []string

Extensions is used to tokenize snippets in directories using the list of file extensions.

func (Extensions) ReadLines

func (e Extensions) ReadLines(dirs ...string) apoco.StreamFunc

ReadLines returns a stream function that reads snippet files (identyfied by the given file extensions) and returns a stream of line tokens.

If a extension ends with `.txt`, one line is read from the text file (no confidences); if the file ends with `.json`, calamari's extended data format is assumed. Otherwise the file is read as a TSV file expecting one char and its confidence on each line.

func (Extensions) TokenizeLines

func (e Extensions) TokenizeLines(ctx context.Context, in <-chan apoco.Token, out chan<- apoco.Token) error

TokenizeLines is a stream function that tokenizes and aligns line tokens.