Versions in this module Expand all Collapse all v0 v0.7.0 Feb 27, 2018 v0.6.0 Jan 5, 2018 Changes in this version + type CharacterTokenizer struct + func NewCharacterTokenizer(f IsTokenRune) *CharacterTokenizer + func (c *CharacterTokenizer) Tokenize(input []byte) analysis.TokenStream + type IsTokenRune func(r rune) bool
Go to main page