tokenizer

Documentation

Index

• type DictTokenizer
  • func (tokenizer DictTokenizer) Tokenize(fragment string) []TextToken
• type TextToken
• type Tokenizer

Types

type DictTokenizer

type DictTokenizer struct {
	WDict map[string]dicttypes.Word
}

Tokenizes Chinese text using a dictionary

func (DictTokenizer) Tokenize

func (tokenizer DictTokenizer) Tokenize(fragment string) []TextToken

Tokenizes a Chinese text string into words and other terms in the dictionary. If the terms are not found in the dictionary then individual characters will be returned. Compares left to right and right to left greedy methods, taking the one with the least tokens.

type TextToken

type TextToken struct {
	Token     string
	DictEntry dicttypes.Word
	Senses    []dicttypes.WordSense
}

A text token contains the results of tokenizing a string

type Tokenizer

type Tokenizer interface {
	Tokenize(fragment string) []TextToken
}

Tokenizes Chinese text