tokenizer

Documentation

Index

• type Tokenizer
  • func Chain(tokenizers ...Tokenizer) Tokenizer
  • func Words(input string) Tokenizer

Types

type Tokenizer

type Tokenizer interface {
	// Next processes the next token and returns true
	// if there are more tokens to process.
	Next() bool
	// Token returns the current token.
	// It must be called after Next.
	Token() string
}

Tokenizer is an interface for tokenizers.

func Chain

func Chain(tokenizers ...Tokenizer) Tokenizer

Chain returns a Tokenizer that combines the given tokenizers into a chain.

The Next method of chain gets tokens serially from the chained tokenizers until the last one returns false. For example, the following code:

ch := tokenizer.Chain(tokenizer.Words("hello world"), tokenizer.Words("this is me"))
for ch.Next() {
	fmt.Printf("%s, ", ch.Token())
}

will print:

hello, world, this, is, me,

func Words

func Words(input string) Tokenizer

Words returns a Tokenizer that tokenizes input into words.