util

Documentation

Index

• func Normalize(word string, noStopWords bool) (string, bool)
• func Sanitize(strs [][]byte, noStopWords bool) []string
• func SortTagItems(items []*model.Tag)
• func SplitToSentences(text []byte) [][]byte
• func TFIDF(t *model.Tag) float64

Functions

func Normalize

func Normalize(word string, noStopWords bool) (string, bool)

Normalize sanitizes word and tells whether it is allowed token or not.

func Sanitize

func Sanitize(strs [][]byte, noStopWords bool) []string

Sanitize ...

func SortTagItems

func SortTagItems(items []*model.Tag)

SortTagItems sorts items by score in descending order, if scores are equal it sorts by count if counts are equal, it sorts string values alphabetically.

func SplitToSentences

func SplitToSentences(text []byte) [][]byte

SplitToSentences splits given text into slice of sentences.

func TFIDF

func TFIDF(t *model.Tag) float64

TFIDF applies TF-IDF to given Tag