tokenizer

Documentation

Overview

Package tokenizer provides utilities for tokenizing text.

Index

• type Cohere
  • func NewCohere(modelName string) (*Cohere, error)
  • func (t *Cohere) GetNumTokens(text string) (uint, error)
  • func (t *Cohere) GetNumTokensFromMessage(messages schema.ChatMessages) (uint, error)
  • func (t *Cohere) GetTokenIDs(text string) ([]uint, error)
• type GPT2
  • func NewGPT2() (*GPT2, error)
  • func (t *GPT2) GetNumTokens(text string) (uint, error)
  • func (t *GPT2) GetNumTokensFromMessage(messages schema.ChatMessages) (uint, error)
  • func (t *GPT2) GetTokenIDs(text string) ([]uint, error)
• type OpenAI
  • func NewOpenAI(modelName string) *OpenAI
  • func (t *OpenAI) GetNumTokens(text string) (uint, error)
  • func (t *OpenAI) GetNumTokensFromMessage(messages schema.ChatMessages) (uint, error)
  • func (t *OpenAI) GetTokenIDs(text string) ([]uint, error)

Types

type Cohere

type Cohere struct {
	// contains filtered or unexported fields
}

func NewCohere

func NewCohere(modelName string) (*Cohere, error)

func (*Cohere) GetNumTokens

func (t *Cohere) GetNumTokens(text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*Cohere) GetNumTokensFromMessage

func (t *Cohere) GetNumTokensFromMessage(messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*Cohere) GetTokenIDs

func (t *Cohere) GetTokenIDs(text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type GPT2

type GPT2 struct {
	// contains filtered or unexported fields
}

func NewGPT2

func NewGPT2() (*GPT2, error)

func (*GPT2) GetNumTokens

func (t *GPT2) GetNumTokens(text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*GPT2) GetNumTokensFromMessage

func (t *GPT2) GetNumTokensFromMessage(messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*GPT2) GetTokenIDs

func (t *GPT2) GetTokenIDs(text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type OpenAI

type OpenAI struct {
	// contains filtered or unexported fields
}

func NewOpenAI

func NewOpenAI(modelName string) *OpenAI

func (*OpenAI) GetNumTokens

func (t *OpenAI) GetNumTokens(text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*OpenAI) GetNumTokensFromMessage

func (t *OpenAI) GetNumTokensFromMessage(messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*OpenAI) GetTokenIDs

func (t *OpenAI) GetTokenIDs(text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.