Versions in this module Expand all Collapse all v0 v0.6.0 Jan 5, 2018 Changes in this version + const Name + func SingleTokenTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error) + type SingleTokenTokenizer struct + func NewSingleTokenTokenizer() *SingleTokenTokenizer + func (t *SingleTokenTokenizer) Tokenize(input []byte) analysis.TokenStream
Go to main page