Versions in this module Expand all Collapse all v2 v2.4.3 Mar 29, 2024 v2.4.2 Mar 29, 2024 Changes in this version + type CharacterTokenizer struct + func NewCharacterTokenizer(f IsTokenRune) *CharacterTokenizer + func (c *CharacterTokenizer) Tokenize(input []byte) analysis.TokenStream + type IsTokenRune func(r rune) bool