tiktoken_loader

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 27, 2023 License: MIT Imports: 5 Imported by: 8

README

tiktoken-go-loader

Alternative bpe loader for tiktoken-go

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type OfflineLoader 

type OfflineLoader struct{}

func NewOfflineLoader 

func NewOfflineLoader() *OfflineLoader

func (*OfflineLoader) LoadTiktokenBpe 

func (l *OfflineLoader) LoadTiktokenBpe(tiktokenBpeFile string) (map[string]int, error)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.