ja

Documentation

Index

• type Filter
  • func NewFilter(opts ...FilterOption) (*Filter, error)
  • func (f Filter) Drop(tokens *[]tokenizer.Token)
  • func (f Filter) Yield(tokens []tokenizer.Token) []string
• type FilterOption
  • func BaseFormFilterOption(p []filter.POS) FilterOption
  • func StopTagsFilterOption(p []filter.POS) FilterOption
  • func StopWordsFilterOption(p []string) FilterOption

Types

type Filter

type Filter struct {
	// contains filtered or unexported fields
}

Filter represents a japanese token filter.

func NewFilter

func NewFilter(opts ...FilterOption) (*Filter, error)

NewFilter returns a filter with the settings commonly used in lucene. To customize, set the options and overwrite the filter.

func (Filter) Drop

func (f Filter) Drop(tokens *[]tokenizer.Token)

Drop drops a token given the provided match function (stop-tags and stop-words).

func (Filter) Yield

func (f Filter) Yield(tokens []tokenizer.Token) []string

Yield returns a filtered word sequence from a token sequence.

type FilterOption

type FilterOption func(*Filter)

FilterOption represents an option of the japanese token filter.

func BaseFormFilterOption

func BaseFormFilterOption(p []filter.POS) FilterOption

BaseFormFilterOption returns a base form filter option.

func StopTagsFilterOption

func StopTagsFilterOption(p []filter.POS) FilterOption

StopTagsFilterOption returns a stop tags filter option.

func StopWordsFilterOption

func StopWordsFilterOption(p []string) FilterOption

StopWordsFilterOption returns a stop words filter option.