Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security Policy
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

pretokenizedstring

package

Go to main page

Versions in this module

v0

v0.2.0

Dec 12, 2020

Changes in this version

type PreTokenizedString

+ func (p *PreTokenizedString) IntoEncoding(wordIndex int, typeID int) (*encodings.Encoding, error)

v0.1.0

Dec 11, 2020

Changes in this version

+ type NormalizedByteSplit struct

+ Offsets strutils.ByteOffsets

+ String string

+ Tokens *[]models.Token

+ type OriginalByteSplit struct

+ Offsets strutils.ByteOffsets

+ String string

+ Tokens *[]models.Token

+ type PreTokenizedString struct

+ func FromNormalizedString(ns *normalizedstring.NormalizedString) *PreTokenizedString

+ func FromString(s string) *PreTokenizedString

+ func (p *PreTokenizedString) GetNormalizedByteSplits() []NormalizedByteSplit

+ func (p *PreTokenizedString) GetOriginalByteSplits() []OriginalByteSplit

+ func (p *PreTokenizedString) Normalize(normalize func(ns *normalizedstring.NormalizedString) error) error

+ func (p *PreTokenizedString) Split(splitFunc SplitFunc) error

+ func (p *PreTokenizedString) Splits() []Split

+ func (p *PreTokenizedString) Tokenize(tokenize func(ns *normalizedstring.NormalizedString) ([]models.Token, error)) error

+ type Split struct

+ NormalizedString *normalizedstring.NormalizedString

+ Tokens *[]models.Token

+ func SplitsFromNormalizedStrings(nss []*normalizedstring.NormalizedString) []Split

+ type SplitFunc func(index int, ns *normalizedstring.NormalizedString) ([]Split, error)