tokenizer

Documentation

Index

• type EnglishTokenizer
  • func NewEnglish(input string) *EnglishTokenizer
  • func (e *EnglishTokenizer) GetTokens() []string
• type Tokenizer

Types

type EnglishTokenizer

type EnglishTokenizer struct {
	// contains filtered or unexported fields
}

func NewEnglish

func NewEnglish(input string) *EnglishTokenizer

New creates a new EnglishTokenizer to be used

func (*EnglishTokenizer) GetTokens

func (e *EnglishTokenizer) GetTokens() []string

type Tokenizer

type Tokenizer interface {
	GetTokens() []string
}