tok

Documentation

Overview

Package tok is a wrapper around ICU boundary iterating functions.

Index

• type Tokenizer
  • func NewTokenizer(s []byte) (*Tokenizer, error)
  • func (t *Tokenizer) Destroy()
  • func (t *Tokenizer) Next() []byte
  • func (t *Tokenizer) Tokens() []string

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer wraps the Tokenizer object in icuc.c.

func NewTokenizer

func NewTokenizer(s []byte) (*Tokenizer, error)

NewTokenizer creates a new Tokenizer object from a given input string of bytes.

func (*Tokenizer) Destroy

func (t *Tokenizer) Destroy()

Destroy destroys the tokenizer object.

func (*Tokenizer) Next

func (t *Tokenizer) Next() []byte

Next returns the next token. It will allocate memory for the token.

func (*Tokenizer) Tokens

func (t *Tokenizer) Tokens() []string

Tokens returns all tokens. If we fail, we return nil.