ngram

README

Ngram for Golang

Branch	Status	Coverage
master

an n-gram is a contiguous sequence of n items from a given sequence of text or speech.

import (
    "fmt"

    "github.com/euskadi31/go-ngram"
)

func main() {
    tokens := ngram.New("Hello", ngram.Bigram).Tokenize()

    fmt.Printf("%v\n", tokens) // ["He", "el", "ll", "lo"]
}

License

go-ngram is licensed under the MIT license.

Documentation

Index

• type SizeType
• type Tokenizer
  • func New(content string, size SizeType) Tokenizer

Types

type SizeType

type SizeType int

SizeType type

const (
	Unigram SizeType = 1
	Bigram  SizeType = 2
	Trigram SizeType = 3
)

N-Gram size type

type Tokenizer

type Tokenizer interface {
	Tokenize() []string
}

Tokenizer interface

func New

func New(content string, size SizeType) Tokenizer

New N-gram Tokenizer