tokenizer

Documentation

Overview

Package tokenizer provides a way to extract tokens (words) from a sentence.

Index

• func Tokens(s string) []string
• func TokensReuseSlice(s string, buf []rune) ([]string, []rune)
• func Unique(tokens []string) []string
• func UniqueInPlace(tokens []string) []string

Functions

func Tokens

func Tokens(s string) []string

Tokens returns lowercase tokens (words) containing in s.

func TokensReuseSlice

func TokensReuseSlice(s string, buf []rune) ([]string, []rune)

TokensReuseSlice returns lowercase tokens (words) containing in s.

func Unique

func Unique(tokens []string) []string

Unique returns a slice of sorted unique tokens.

func UniqueInPlace

func UniqueInPlace(tokens []string) []string

UniqueInPlace returns a slice of sorted unique tokens, reusing the tokens slice.