tokenizer

package
v0.0.111 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 16, 2024 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package tokenizer provides utilities for tokenizing text.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Claude added in v0.0.47

type Claude struct {
	// contains filtered or unexported fields
}

func NewClaude added in v0.0.47

func NewClaude() (*Claude, error)

func (*Claude) GetNumTokens added in v0.0.47

func (t *Claude) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*Claude) GetNumTokensFromMessage added in v0.0.47

func (t *Claude) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*Claude) GetTokenIDs added in v0.0.47

func (t *Claude) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type Cohere added in v0.0.30

type Cohere struct {
	// contains filtered or unexported fields
}

func NewCohere added in v0.0.30

func NewCohere(modelName string) (*Cohere, error)

func (*Cohere) GetNumTokens added in v0.0.30

func (t *Cohere) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*Cohere) GetNumTokensFromMessage added in v0.0.30

func (t *Cohere) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*Cohere) GetTokenIDs added in v0.0.30

func (t *Cohere) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type GPT2 added in v0.0.24

type GPT2 struct {
	// contains filtered or unexported fields
}

func NewGPT2 added in v0.0.24

func NewGPT2() (*GPT2, error)

func (*GPT2) GetNumTokens added in v0.0.24

func (t *GPT2) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*GPT2) GetNumTokensFromMessage added in v0.0.24

func (t *GPT2) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*GPT2) GetTokenIDs added in v0.0.24

func (t *GPT2) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

type GoogleGenAI added in v0.0.92

type GoogleGenAI struct {
	// contains filtered or unexported fields
}

func NewGoogleGenAI added in v0.0.92

func NewGoogleGenAI(client GoogleGenAIClient, model string) *GoogleGenAI

func (*GoogleGenAI) GetNumTokens added in v0.0.92

func (t *GoogleGenAI) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*GoogleGenAI) GetNumTokensFromMessage added in v0.0.92

func (t *GoogleGenAI) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

type GoogleGenAIClient added in v0.0.92

type GoogleGenAIClient interface {
	CountTokens(context.Context, *generativelanguagepb.CountTokensRequest, ...gax.CallOption) (*generativelanguagepb.CountTokensResponse, error)
}

GoogleGenAIClient is an interface for the GoogleGenAI model client.

type OpenAI

type OpenAI struct {
	// contains filtered or unexported fields
}

func NewOpenAI

func NewOpenAI(modelName string) *OpenAI

func (*OpenAI) GetNumTokens

func (t *OpenAI) GetNumTokens(ctx context.Context, text string) (uint, error)

GetNumTokens returns the number of tokens in the provided text.

func (*OpenAI) GetNumTokensFromMessage

func (t *OpenAI) GetNumTokensFromMessage(ctx context.Context, messages schema.ChatMessages) (uint, error)

GetNumTokensFromMessage returns the number of tokens in the provided chat messages.

func (*OpenAI) GetTokenIDs

func (t *OpenAI) GetTokenIDs(ctx context.Context, text string) ([]uint, error)

GetTokenIDs returns the token IDs corresponding to the provided text.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL