Versions in this module Expand all Collapse all v0 v0.8.1 Sep 20, 2019 Changes in this version + type CharacterTokenizer struct + func NewCharacterTokenizer(f IsTokenRune) *CharacterTokenizer + func (c *CharacterTokenizer) Tokenize(input []byte) analysis.TokenStream + type IsTokenRune func(r rune) bool
Go to main page