tokenizer

README

jieba-go

A copy-cat implementation of jieba as a learning exercise. Only the tokenizer is implemented.

Documentation

Index

• type Tokenizer
  • func NewJiebaTokenizer() *Tokenizer
  • func NewTokenizer(dictionaryFile string) *Tokenizer
  • func (tk *Tokenizer) AddWord(word string, freq int)
  • func (tk *Tokenizer) Cut(text string, useHmm bool) []string
  • func (tk *Tokenizer) CutParallel(text string, hmm bool, numWorkers int, ordered bool) []string

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

func NewJiebaTokenizer

func NewJiebaTokenizer() *Tokenizer

func NewTokenizer

func NewTokenizer(dictionaryFile string) *Tokenizer

func (*Tokenizer) AddWord

func (tk *Tokenizer) AddWord(word string, freq int)

Add a word to the prefix dictionary. If word already exists, the word's frequency value will be updated. If freq is less than 1, a frequency will be automatically calculated.

func (*Tokenizer) Cut

func (tk *Tokenizer) Cut(text string, useHmm bool) []string

Cut text and return a slice of tokens.

func (*Tokenizer) CutParallel

func (tk *Tokenizer) CutParallel(text string, hmm bool, numWorkers int, ordered bool) []string

Perform Cut in worker goroutines in parallel. If ordered is true, the returned slice will be sorted according to the order of the input text. Sorting will adversely impact performance by approximately 30%.