tok

Documentation

Overview

Package tok is a wrapper around ICU boundary iterating functions.

Index

• func ICUDisabled() bool
• type Tokenizer
  • func NewTokenizer(s []byte) (*Tokenizer, error)
  • func (t *Tokenizer) Destroy()
  • func (t *Tokenizer) Next() []byte
  • func (t *Tokenizer) Tokens() []string

Functions

func ICUDisabled

func ICUDisabled() bool

ICUDisabled returns whether ICU is disabled. It may be disabled if we are in embed mode and no data file is specified.

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer wraps the Tokenizer object in icuc.c.

func NewTokenizer

func NewTokenizer(s []byte) (*Tokenizer, error)

NewTokenizer creates a new Tokenizer object from a given input string of bytes.

func (*Tokenizer) Destroy

func (t *Tokenizer) Destroy()

Destroy destroys the tokenizer object.

func (*Tokenizer) Next

func (t *Tokenizer) Next() []byte

Next returns the next token. It will allocate memory for the token.

func (*Tokenizer) Tokens

func (t *Tokenizer) Tokens() []string

Tokens returns all tokens. If we fail, we return nil.