Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type WhiteSpaceSplitPreTokenizer ¶
type WhiteSpaceSplitPreTokenizer struct{}
WhiteSpaceSplitPreTokenizer allows the generation of pre-tokens splitting by whitespace-like characters.
func (*WhiteSpaceSplitPreTokenizer) PreTokenize ¶
func (w *WhiteSpaceSplitPreTokenizer) PreTokenize(pts *pretokenizedstring.PreTokenizedString) error
PreTokenize splits the NormalizedString by whitespace-like characters
Source Files
Click to show internal directories.
Click to hide internal directories.