pos

Documentation

Overview

Package pos model data The data from https://github.com/fxsjy/jieba

Index

• type Dict
  • func (d *Dict) AddToken(text string, freq float64, pos ...string) error
  • func (d *Dict) Freq(key string) (float64, string, bool)
  • func (d *Dict) Pos(key string) (string, bool)
  • func (d *Dict) RemoveToken(text string) error
• type SegPos
• type Segmenter
  • func (seg *Segmenter) Cut(text string, hmm ...bool) (result []gse.SegPos)
  • func (seg *Segmenter) LoadDict(fileName ...string) error
  • func (seg *Segmenter) Trim(se []gse.SegPos) (re []gse.SegPos)
  • func (seg *Segmenter) TrimPunct(se []gse.SegPos) (re []gse.SegPos)
  • func (seg *Segmenter) TrimWithPos(se []gse.SegPos, pos ...string) (re []gse.SegPos)
  • func (seg *Segmenter) WithGse(segs gse.Segmenter)

Types

type Dict

type Dict struct {
	Seg gse.Segmenter
	// contains filtered or unexported fields
}

Dict type a dictionary for the word segmentation.

func (*Dict) AddToken

func (d *Dict) AddToken(text string, freq float64, pos ...string) error

AddToken add new text to token

func (*Dict) Freq

func (d *Dict) Freq(key string) (float64, string, bool)

Freq find the word return the frequency and existenced

func (*Dict) Pos

func (d *Dict) Pos(key string) (string, bool)

Pos find the key return the POS and existenced

func (*Dict) RemoveToken

func (d *Dict) RemoveToken(text string) error

RemoveToken remove the token in the dict

type SegPos

type SegPos struct {
	Text, Pos string
}

SegPos type POS struct

type Segmenter

type Segmenter struct {
	// contains filtered or unexported fields
}

Segmenter is a segmentation struct

func (*Segmenter) Cut

func (seg *Segmenter) Cut(text string, hmm ...bool) (result []gse.SegPos)

Cut cuts a text into words. Parameter hmm controls whether to use the HMM

func (*Segmenter) LoadDict

func (seg *Segmenter) LoadDict(fileName ...string) error

LoadDict load dictionary from the file.

func (*Segmenter) Trim

func (seg *Segmenter) Trim(se []gse.SegPos) (re []gse.SegPos)

Trim not space and punct

func (*Segmenter) TrimPunct

func (seg *Segmenter) TrimPunct(se []gse.SegPos) (re []gse.SegPos)

TrimPunct not space and punct

func (*Segmenter) TrimWithPos

func (seg *Segmenter) TrimWithPos(se []gse.SegPos, pos ...string) (re []gse.SegPos)

TrimWithPos trim some pos

func (*Segmenter) WithGse

func (seg *Segmenter) WithGse(segs gse.Segmenter)

WithGse register the gse segmenter