token

Documentation

Overview

Package token tokenizes data for dupi.

Index

• type Config
  • func DefaultConfig() *Config
• type T
  • func Tokenize(dst []T, d []byte, offset uint32) []T
  • func (t *T) String() string
• type Tag
  • func (t Tag) String() string
• type TokenizerFunc
  • func FromConfig(cfg *Config) (TokenizerFunc, error)

Types

type Config

type Config struct {
	Name string
}

Config describes tokenizer configurations.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns the default tokenizer config for dupy.

type T

type T struct {
	Tag Tag
	Lit []byte
	Pos uint32
}

Type T represents a token.

func Tokenize

func Tokenize(dst []T, d []byte, offset uint32) []T

Tokenize is a tokenizer function.

func (*T) String

func (t *T) String() string

type Tag

type Tag int

Tag represents a value in an enumeration of values to associate with a token.

const (
	Word Tag = 35 + iota
	Other
	Eod
)

func (Tag) String

func (t Tag) String() string

type TokenizerFunc

type TokenizerFunc func(dst []T, dat []byte, offset uint32) []T

TokenizerFunc is the type of a function used for tokenizing document data.

func FromConfig

func FromConfig(cfg *Config) (TokenizerFunc, error)

FromConfig attempts to create a tokenizer function from a configuration.