p50kbase

Documentation

Overview

Package p50kbase registers the "p50k_base" tokenizer with gotoken. To use this tokenizer:

import (
    "github.com/peterheb/gotoken"
    _ "github.com/peterheb/gotoken/p50kbase"
)
...
tok, err := gotoken.GetTokenizer("p50k_base")

This file was generated from the following data:

• Source URL: https://openaipublic.blob.core.windows.net/encodings/p50k_base.tiktoken
• Source SHA-256: 94b5ca7dff4d00767bc256fdd1b27e5b17361d7b8a5f968547f9f23eb70d2069
• Generated: 2023-04-16T22:17:01Z

Index

• Constants

Constants

const (
	EndOfText = "<|endoftext|>"
	FIMPrefix = "<|fim_prefix|>"
	FIMMiddle = "<|fim_middle|>"
	FIMSuffix = "<|fim_suffix|>"
)

These special tokens are defined by this encoding.