etokenizer

Documentation

Index

• type Tokenizer
  • func AlphaNumCaseTokenizer() Tokenizer

Types

type Tokenizer

type Tokenizer interface {
	Tokens(s string) []string
}

Tokenizer break a string into tokens.

func AlphaNumCaseTokenizer

func AlphaNumCaseTokenizer() Tokenizer

AlphaNumCaseTokenizer returns alpha-numeric tokens. This tokenizer ignores characters except alpha-numeric. This tokenizer splits token on case change. For example, "Powered by GoLang version1.17." is tokenized to "Powered", "by", "Go", "Lang", "version1", and "17".