jieba

Documentation

Index

• Constants
• func AnalyzerJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (*analysis.Analyzer, error)
• func CheckDictFiles(dict string) error
• func JiebaTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)
• func NewIndexWithJieba(findex string, dictPath string, searchmode bool, readOnly bool) (bleve.Index, error)
• func NewJiebaTokenizer(dictFilePath, hmm, userDictPath, idfDict, stopDict string, searchMode bool) (analysis.Tokenizer, error)
• func NewMemIndexWithJieba(dictPath string, searchmode bool) (bleve.Index, error)
• func StopTokenFilterJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenFilter, error)
• func TokenMapJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenMap, error)
• type JiebaTokenizer
  • func (jt *JiebaTokenizer) Tokenize(input []byte) analysis.TokenStream

Constants

const (
	JiebaName = "jieba"
)

JiebaName is the jieba analyzer/tokenizer name.

Functions

func AnalyzerJiebaConstructor

func AnalyzerJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (*analysis.Analyzer, error)

func CheckDictFiles

func CheckDictFiles(dict string) error

func JiebaTokenizerConstructor

func JiebaTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)

func NewIndexWithJieba

func NewIndexWithJieba(findex string, dictPath string, searchmode bool, readOnly bool) (bleve.Index, error)

func NewJiebaTokenizer

func NewJiebaTokenizer(dictFilePath, hmm, userDictPath, idfDict, stopDict string, searchMode bool) (analysis.Tokenizer, error)

func NewMemIndexWithJieba

func NewMemIndexWithJieba(dictPath string, searchmode bool) (bleve.Index, error)

func StopTokenFilterJiebaConstructor

func StopTokenFilterJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenFilter, error)

func TokenMapJiebaConstructor

func TokenMapJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenMap, error)

TokenMapConstructor create a stop word token map. Parameter config can contains following parameters:

stopdict_path: optional, user stop dict file path

Types

type JiebaTokenizer

type JiebaTokenizer struct {
	// contains filtered or unexported fields
}

JiebaTokenizer is the beleve tokenizer for jiebago.

func (*JiebaTokenizer) Tokenize

func (jt *JiebaTokenizer) Tokenize(input []byte) analysis.TokenStream

Tokenize cuts input into bleve token stream.