tiktoken

Documentation

Overview

Package tiktoken implements a wrapper around the github.com/tiktoken-go/tokenizer library. It is compatible with OpenAI's tiktoken Python library.

Index

• func Len(forModel string) func(string) int
• func Splitter(model string, options flowllm.SplitterOptions) flowllm.Splitter

Functions

func Len

func Len(forModel string) func(string) int

Len returns a len function for the given model that returns the number of tokens in a string.

func Splitter

func Splitter(model string, options flowllm.SplitterOptions) flowllm.Splitter