tokenizer

Documentation

Index

• type Tokenizer
  • func NewTokenizer(opt ...string) *Tokenizer
  • func (m *Tokenizer) Tokenize(text string) Tokens
• type Tokens
  • func (m Tokens) DistinctByNoun() (tokens Tokens)
  • func (m Tokens) Len() int
  • func (m Tokens) Less(i, j int) bool
  • func (m Tokens) Sort() Tokens
  • func (m Tokens) Swap(i, j int)

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer is kagome.Tokenizer wrapper

func NewTokenizer

func NewTokenizer(opt ...string) *Tokenizer

NewTokenizer initialize kagome.Tokenizer

func (*Tokenizer) Tokenize

func (m *Tokenizer) Tokenize(text string) Tokens

Tokenize create []kagome.Token

type Tokens

type Tokens []tokenizer.Token

Tokens is tokenizer.Token wrapper

func (Tokens) DistinctByNoun

func (m Tokens) DistinctByNoun() (tokens Tokens)

DistinctByNoun return Tokens distinct by nouns

func (Tokens) Len

func (m Tokens) Len() int

Len is implementation of sort.Sort

func (Tokens) Less

func (m Tokens) Less(i, j int) bool

Less is implementation of sort.Sort

func (Tokens) Sort

func (m Tokens) Sort() Tokens

Sort as kagome.Token.Surface ASC

func (Tokens) Swap

func (m Tokens) Swap(i, j int)

Swap is implementation of sort.Sort