tok

package
v0.7.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 27, 2017 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package tok is a wrapper around ICU boundary iterating functions.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ICUDisabled added in v0.7.2

func ICUDisabled() bool

ICUDisabled returns whether ICU is disabled. It may be disabled if we are in embed mode and no data file is specified.

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer wraps the Tokenizer object in icuc.c.

func NewTokenizer

func NewTokenizer(s []byte) (*Tokenizer, error)

NewTokenizer creates a new Tokenizer object from a given input string of bytes.

func (*Tokenizer) Destroy

func (t *Tokenizer) Destroy()

Destroy destroys the tokenizer object.

func (*Tokenizer) Next

func (t *Tokenizer) Next() []byte

Next returns the next token. It will allocate memory for the token.

func (*Tokenizer) Tokens

func (t *Tokenizer) Tokens() []string

Tokens returns all tokens. If we fail, we return nil.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL