bpetokenizer

Imports

• github.com/nlpodyssey/gotokenizers/encodings
• github.com/nlpodyssey/gotokenizers/models
• github.com/nlpodyssey/gotokenizers/models/bpemodel
• github.com/nlpodyssey/gotokenizers/normalizedstring
• github.com/nlpodyssey/gotokenizers/pretokenizedstring
• github.com/nlpodyssey/gotokenizers/pretokenizers/bytelevelpretokenizer
• github.com/nlpodyssey/gotokenizers/strutils
• github.com/nlpodyssey/gotokenizers/vocabulary

Imports in module “github.com/nlpodyssey/cybertron”

• github.com/nlpodyssey/cybertron/pkg/tokenizers

Standard library imports

• fmt
• path/filepath
• strings