snippets

Documentation

Index

• type Extensions
  • func (e Extensions) Tokenize(dirs ...string) apoco.StreamFunc

Types

type Extensions

type Extensions []string

Extensions is used to tokenize snippets in directories using the list of file extensions.

func (Extensions) Tokenize

func (e Extensions) Tokenize(dirs ...string) apoco.StreamFunc

Tokenize tokenizes tokens from line snippets TSV files (identyfied by the given file extensions) and alignes them accordingly. If a extension ends with `.txt`, one line is read from the text file (no confidences). Otherwise the file is read as a TSV file expecting on char and its confidence on each line.