r50kbase

Documentation

Overview

Package r50kbase registers the "r50k_base" tokenizer with gotoken. To use this tokenizer:

import (
    "github.com/peterheb/gotoken"
    _ "github.com/peterheb/gotoken/r50kbase"
)
...
tok, err := gotoken.GetTokenizer("r50k_base")

This file was generated from the following data:

• Source URL: https://openaipublic.blob.core.windows.net/encodings/r50k_base.tiktoken
• Source SHA-256: 306cd27f03c1a714eca7108e03d66b7dc042abe8c258b44c199a7ed9838dd930
• Generated: 2023-04-16T22:16:58Z

Index

• Constants

Constants

const (
	EndOfText = "<|endoftext|>"
)

This special token is defined by this encoding.