gen

README

gotoken generator

This directory contains the generator that converts OpenAI .tiktoken files into data embedded in Go source files.

Usage

From this directory, run go generate. It will output the generated Go source at ../{encoding}/data.go, where {encoding} gets replaced with each of the supported tokenizers.

Documentation

Overview

Package gen generates the data.go files for the gotoken's encoding sub-packages. To run, "go generate". This will overwrite files in ../{encoding}/data.go!

This program generates data structures based on the ".tiktoken" files provided by OpenAI. These files are licensed under the MIT license and include the following notice: Copyright (c) 2022 OpenAI, Shantanu Jain.

• See also: https://github.com/openai/tiktoken/blob/main/LICENSE