tokenizer

package
v0.1.8-alpha Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 20, 2021 License: Apache-2.0 Imports: 5 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type TokenizeFunc

type TokenizeFunc func(text string) []string

TokenizeFunc is a naive tokenizer.

func Tokenizer

func Tokenizer() TokenizeFunc

Tokenizer returns the default tokenize functor.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL