tokenizers

Documentation

Overview

Package tokenizers is an interim solution while developing `gotokenizers` (https://github.com/nlpodyssey/gotokenizers). APIs and implementations may be subject to frequent refactoring.

Index

• func GetStrings(tokens []StringOffsetsPair) []string
• type OffsetsType
  • func GetOffsets(tokens []StringOffsetsPair) []OffsetsType
• type StringOffsetsPair
• type Tokenizer

Functions

func GetStrings

func GetStrings(tokens []StringOffsetsPair) []string

GetStrings returns a sequence of string values from the given slice of StringOffsetsPair.

Types

type OffsetsType

type OffsetsType struct {
	Start int
	End   int
}

OffsetsType represents a (start, end) offsets pair. It usually represents a lower inclusive index position, and an upper exclusive position.

func GetOffsets

func GetOffsets(tokens []StringOffsetsPair) []OffsetsType

GetOffsets returns a sequence of offsets values from the given slice of StringOffsetsPair.

type StringOffsetsPair

type StringOffsetsPair struct {
	String  string
	Offsets OffsetsType
}

StringOffsetsPair represents a string value paired with offsets bounds. It usually represents a token string and its offsets positions in the original string.

type Tokenizer

type Tokenizer interface {
	Tokenize(text string) []StringOffsetsPair
}

Tokenizer is implemented by any value that has the Tokenize method.