tok

package
v0.9.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 4, 2017 License: AGPL-3.0, Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	FTSTokenizerName = "fulltext"
)

Variables

This section is empty.

Functions

func BuildTokens added in v0.9.0 

func BuildTokens(val interface{}, t Tokenizer) ([]string, error)

BuildTokens tokenizes a value, creating strings that can be used to create index keys.

func EncodeGeoTokens added in v0.8.2 

func EncodeGeoTokens(tokens []string)

func EncodeRegexTokens added in v0.8.2 

func EncodeRegexTokens(tokens []string)

func FtsTokenizerName added in v0.8.2 

func FtsTokenizerName(lang string) string

func GetTextTokens added in v0.8.2 

func GetTextTokens(funcArgs []string, lang string) ([]string, error)

func GetTokens added in v0.8.2 

func GetTokens(funcArgs []string) ([]string, error)

func LoadCustomTokenizer added in v0.9.0 

func LoadCustomTokenizer(soFile string)

Types

type BoolTokenizer added in v0.8.2 

type BoolTokenizer struct{}

func (BoolTokenizer) Identifier added in v0.8.2 

func (t BoolTokenizer) Identifier() byte

func (BoolTokenizer) IsLossy added in v0.8.2 

func (t BoolTokenizer) IsLossy() bool

func (BoolTokenizer) IsSortable added in v0.8.2 

func (t BoolTokenizer) IsSortable() bool

func (BoolTokenizer) Name added in v0.8.2 

func (t BoolTokenizer) Name() string

func (BoolTokenizer) Tokens added in v0.8.2 

func (t BoolTokenizer) Tokens(v interface{}) ([]string, error)

func (BoolTokenizer) Type added in v0.8.2 

func (t BoolTokenizer) Type() string

type CustomTokenizer added in v0.9.0 

type CustomTokenizer struct {
	PluginTokenizer
}

func (CustomTokenizer) IsLossy added in v0.9.0 

func (CustomTokenizer) IsLossy() bool

func (CustomTokenizer) IsSortable added in v0.9.0 

func (CustomTokenizer) IsSortable() bool

It doesn't make sense for plugins to implement the following methods, so they're hardcoded.

type DayTokenizer added in v0.8.2 

type DayTokenizer struct{}

func (DayTokenizer) Identifier added in v0.8.2 

func (t DayTokenizer) Identifier() byte

func (DayTokenizer) IsLossy added in v0.8.2 

func (t DayTokenizer) IsLossy() bool

func (DayTokenizer) IsSortable added in v0.8.2 

func (t DayTokenizer) IsSortable() bool

func (DayTokenizer) Name added in v0.8.2 

func (t DayTokenizer) Name() string

func (DayTokenizer) Tokens added in v0.8.2 

func (t DayTokenizer) Tokens(v interface{}) ([]string, error)

func (DayTokenizer) Type added in v0.8.2 

func (t DayTokenizer) Type() string

type ExactTokenizer added in v0.7.3 

type ExactTokenizer struct{}

func (ExactTokenizer) Identifier added in v0.8.2 

func (t ExactTokenizer) Identifier() byte

func (ExactTokenizer) IsLossy added in v0.8.2 

func (t ExactTokenizer) IsLossy() bool

func (ExactTokenizer) IsSortable added in v0.8.2 

func (t ExactTokenizer) IsSortable() bool

func (ExactTokenizer) Name added in v0.7.3 

func (t ExactTokenizer) Name() string

func (ExactTokenizer) Tokens added in v0.7.3 

func (t ExactTokenizer) Tokens(v interface{}) ([]string, error)

func (ExactTokenizer) Type added in v0.7.3 

func (t ExactTokenizer) Type() string

type FloatTokenizer added in v0.7.3 

type FloatTokenizer struct{}

func (FloatTokenizer) Identifier added in v0.8.2 

func (t FloatTokenizer) Identifier() byte

func (FloatTokenizer) IsLossy added in v0.8.2 

func (t FloatTokenizer) IsLossy() bool

func (FloatTokenizer) IsSortable added in v0.8.2 

func (t FloatTokenizer) IsSortable() bool

func (FloatTokenizer) Name added in v0.7.3 

func (t FloatTokenizer) Name() string

func (FloatTokenizer) Tokens added in v0.7.3 

func (t FloatTokenizer) Tokens(v interface{}) ([]string, error)

func (FloatTokenizer) Type added in v0.7.3 

func (t FloatTokenizer) Type() string

type FullTextTokenizer added in v0.8.2 

type FullTextTokenizer struct {
	Lang string
}

Full text tokenizer, with language support

func (FullTextTokenizer) Identifier added in v0.8.2 

func (t FullTextTokenizer) Identifier() byte

func (FullTextTokenizer) IsLossy added in v0.8.2 

func (t FullTextTokenizer) IsLossy() bool

func (FullTextTokenizer) IsSortable added in v0.8.2 

func (t FullTextTokenizer) IsSortable() bool

func (FullTextTokenizer) Name added in v0.8.2 

func (t FullTextTokenizer) Name() string

func (FullTextTokenizer) Tokens added in v0.8.2 

func (t FullTextTokenizer) Tokens(v interface{}) ([]string, error)

func (FullTextTokenizer) Type added in v0.8.2 

func (t FullTextTokenizer) Type() string

type GeoTokenizer added in v0.7.3 

type GeoTokenizer struct{}

func (GeoTokenizer) Identifier added in v0.8.2 

func (t GeoTokenizer) Identifier() byte

func (GeoTokenizer) IsLossy added in v0.8.2 

func (t GeoTokenizer) IsLossy() bool

func (GeoTokenizer) IsSortable added in v0.8.2 

func (t GeoTokenizer) IsSortable() bool

func (GeoTokenizer) Name added in v0.7.3 

func (t GeoTokenizer) Name() string

func (GeoTokenizer) Tokens added in v0.7.3 

func (t GeoTokenizer) Tokens(v interface{}) ([]string, error)

func (GeoTokenizer) Type added in v0.7.3 

func (t GeoTokenizer) Type() string

type HashTokenizer added in v0.8.2 

type HashTokenizer struct{}

func (HashTokenizer) Identifier added in v0.8.2 

func (t HashTokenizer) Identifier() byte

func (HashTokenizer) IsLossy added in v0.8.2 

func (t HashTokenizer) IsLossy() bool

func (HashTokenizer) IsSortable added in v0.8.2 

func (t HashTokenizer) IsSortable() bool

func (HashTokenizer) Name added in v0.8.2 

func (t HashTokenizer) Name() string

func (HashTokenizer) Tokens added in v0.8.2 

func (t HashTokenizer) Tokens(v interface{}) ([]string, error)

func (HashTokenizer) Type added in v0.8.2 

func (t HashTokenizer) Type() string

type HourTokenizer added in v0.8.2 

type HourTokenizer struct{}

func (HourTokenizer) Identifier added in v0.8.2 

func (t HourTokenizer) Identifier() byte

func (HourTokenizer) IsLossy added in v0.8.2 

func (t HourTokenizer) IsLossy() bool

func (HourTokenizer) IsSortable added in v0.8.2 

func (t HourTokenizer) IsSortable() bool

func (HourTokenizer) Name added in v0.8.2 

func (t HourTokenizer) Name() string

func (HourTokenizer) Tokens added in v0.8.2 

func (t HourTokenizer) Tokens(v interface{}) ([]string, error)

func (HourTokenizer) Type added in v0.8.2 

func (t HourTokenizer) Type() string

type IntTokenizer added in v0.8.2 

type IntTokenizer struct{}

func (IntTokenizer) Identifier added in v0.8.2 

func (t IntTokenizer) Identifier() byte

func (IntTokenizer) IsLossy added in v0.8.2 

func (t IntTokenizer) IsLossy() bool

func (IntTokenizer) IsSortable added in v0.8.2 

func (t IntTokenizer) IsSortable() bool

func (IntTokenizer) Name added in v0.8.2 

func (t IntTokenizer) Name() string

func (IntTokenizer) Tokens added in v0.8.2 

func (t IntTokenizer) Tokens(v interface{}) ([]string, error)

func (IntTokenizer) Type added in v0.8.2 

func (t IntTokenizer) Type() string

type MonthTokenizer added in v0.8.2 

type MonthTokenizer struct{}

func (MonthTokenizer) Identifier added in v0.8.2 

func (t MonthTokenizer) Identifier() byte

func (MonthTokenizer) IsLossy added in v0.8.2 

func (t MonthTokenizer) IsLossy() bool

func (MonthTokenizer) IsSortable added in v0.8.2 

func (t MonthTokenizer) IsSortable() bool

func (MonthTokenizer) Name added in v0.8.2 

func (t MonthTokenizer) Name() string

func (MonthTokenizer) Tokens added in v0.8.2 

func (t MonthTokenizer) Tokens(v interface{}) ([]string, error)

func (MonthTokenizer) Type added in v0.8.2 

func (t MonthTokenizer) Type() string

type PluginTokenizer added in v0.9.0 

type PluginTokenizer interface {
	Name() string
	Type() string
	Tokens(value interface{}) ([]string, error)
	Identifier() byte
}

PluginTokenizer is implemented by external plugins loaded dynamically via *.so files. It follows the implementation semantics of the Tokenizer interface.

Think carefully before modifying this interface, as it would break users' plugins.

type TermTokenizer added in v0.7.3 

type TermTokenizer struct{}

func (TermTokenizer) Identifier added in v0.8.2 

func (t TermTokenizer) Identifier() byte

func (TermTokenizer) IsLossy added in v0.8.2 

func (t TermTokenizer) IsLossy() bool

func (TermTokenizer) IsSortable added in v0.8.2 

func (t TermTokenizer) IsSortable() bool

func (TermTokenizer) Name added in v0.7.3 

func (t TermTokenizer) Name() string

func (TermTokenizer) Tokens added in v0.7.3 

func (t TermTokenizer) Tokens(v interface{}) ([]string, error)

func (TermTokenizer) Type added in v0.7.3 

func (t TermTokenizer) Type() string

type Tokenizer 

type Tokenizer interface {

	// Name is name of tokenizer. This should be unique.
	Name() string

	// Type returns the string representation of the typeID that we care about.
	Type() string

	// Tokens return tokens for a given value. The tokens shouldn't be encoded
	// with the byte identifier.
	Tokens(value interface{}) ([]string, error)

	// Identifier returns the prefix byte for this token type. This should be
	// unique. The range 0x80 to 0xff (inclusive) is reserved for user-provided
	// custom tokenizers.
	Identifier() byte

	// IsSortable returns true if the tokenizer can be used for sorting/ordering.
	IsSortable() bool

	// IsLossy() returns true if we don't store the values directly as index keys
	// during tokenization. If a predicate is tokenized using an IsLossy() tokenizer,
	// then we need to fetch the actual value and compare.
	IsLossy() bool
}

Tokenizer defines what a tokenizer must provide.

func GetTokenizer added in v0.7.3 

func GetTokenizer(name string) (Tokenizer, bool)

GetTokenizer returns tokenizer given unique name.

type TrigramTokenizer added in v0.8.2 

type TrigramTokenizer struct{}

func (TrigramTokenizer) Identifier added in v0.8.2 

func (t TrigramTokenizer) Identifier() byte

func (TrigramTokenizer) IsLossy added in v0.8.2 

func (t TrigramTokenizer) IsLossy() bool

func (TrigramTokenizer) IsSortable added in v0.8.2 

func (t TrigramTokenizer) IsSortable() bool

func (TrigramTokenizer) Name added in v0.8.2 

func (t TrigramTokenizer) Name() string

func (TrigramTokenizer) Tokens added in v0.8.2 

func (t TrigramTokenizer) Tokens(v interface{}) ([]string, error)

func (TrigramTokenizer) Type added in v0.8.2 

func (t TrigramTokenizer) Type() string

type YearTokenizer added in v0.8.2 

type YearTokenizer struct{}

func (YearTokenizer) Identifier added in v0.8.2 

func (t YearTokenizer) Identifier() byte

func (YearTokenizer) IsLossy added in v0.8.2 

func (t YearTokenizer) IsLossy() bool

func (YearTokenizer) IsSortable added in v0.8.2 

func (t YearTokenizer) IsSortable() bool

func (YearTokenizer) Name added in v0.8.2 

func (t YearTokenizer) Name() string

func (YearTokenizer) Tokens added in v0.8.2 

func (t YearTokenizer) Tokens(v interface{}) ([]string, error)

func (YearTokenizer) Type added in v0.8.2 

func (t YearTokenizer) Type() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.