nlp

Documentation

Index

• Variables
• func NewNLPTokenizer(credentialsFilename string, lang string) tokenize.Tokenizer
• type NLPTokenizer
  • func (nlp NLPTokenizer) Tokenize(ctx context.Context, text string) ([]tokenize.Token, error)

Variables

var AutoLang string = "auto"

AutoLang tries to automatically recognize the language

var (
	ErrMaxRetries = errors.New("max retries reached")
)

Functions

func NewNLPTokenizer

func NewNLPTokenizer(credentialsFilename string, lang string) tokenize.Tokenizer

NewNLPTokenizer returns a new NLP tokenizer instance. Note that NLPTokenizer has a built-in retrier

Types

type NLPTokenizer

type NLPTokenizer struct {
	// contains filtered or unexported fields
}

NLPTokenizer tokenizes a text using Google NLP

func (NLPTokenizer) Tokenize

func (nlp NLPTokenizer) Tokenize(ctx context.Context, text string) ([]tokenize.Token, error)

Tokenize tokenizes a text