Versions in this module Expand all Collapse all v0 v0.8.1 Sep 20, 2019 Changes in this version + const Name + func UnicodeTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error) + type UnicodeTokenizer struct + func NewUnicodeTokenizer() *UnicodeTokenizer + func (rt *UnicodeTokenizer) Tokenize(input []byte) analysis.TokenStream
Go to main page