llms

Documentation

Overview

Package llms provides unified support for interacting with different Language Models (LLMs) from various providers. Designed with an extensible architecture, the package facilitates seamless integration of LLMs with a focus on modularity, encapsulation, and easy configurability.

The package includes the following subpackages for LLM providers: 1. Hugging Face: llms/huggingface/ 2. Local LLM: llms/local/ 3. OpenAI: llms/openai/ 4. Vertex AI: llms/vertexai/ 5. Cohere: llms/cohere/

Each subpackage includes provider-specific LLM implementations and helper files for communication with supported LLM providers. The internal directories within these subpackages contain provider-specific client and API implementations.

The `llms.go` file contains the types and interfaces for interacting with different LLMs.

The `options.go` file provides various options and functions to configure the LLMs.

Index

• func CalculateMaxTokens(model, text string) int
• func CountTokens(model, text string) int
• func GetModelContextSize(model string) int
• type CallOption
  • func WithFrequencyPenalty(frequencyPenalty float64) CallOption
  • func WithFunctionCallBehavior(behavior FunctionCallBehavior) CallOption
  • func WithFunctions(functions []FunctionDefinition) CallOption
  • func WithMaxLength(maxLength int) CallOption
  • func WithMaxTokens(maxTokens int) CallOption
  • func WithMinLength(minLength int) CallOption
  • func WithModel(model string) CallOption
  • func WithN(n int) CallOption
  • func WithOptions(options CallOptions) CallOption
  • func WithPresencePenalty(presencePenalty float64) CallOption
  • func WithRepetitionPenalty(repetitionPenalty float64) CallOption
  • func WithSeed(seed int) CallOption
  • func WithStopWords(stopWords []string) CallOption
  • func WithStreamingFunc(streamingFunc func(ctx context.Context, chunk []byte) error) CallOption
  • func WithTemperature(temperature float64) CallOption
  • func WithTopK(topK int) CallOption
  • func WithTopP(topP float64) CallOption
• type CallOptions
• type ChatLLM
• type FunctionCallBehavior
• type FunctionDefinition
• type Generation
• type LLM
• type LLMResult
  • func GenerateChatPrompt(ctx context.Context, l ChatLLM, promptValues []schema.PromptValue, ...) (LLMResult, error)
  • func GeneratePrompt(ctx context.Context, l LLM, promptValues []schema.PromptValue, ...) (LLMResult, error)
• type LanguageModel

Functions

func CalculateMaxTokens

func CalculateMaxTokens(model, text string) int

CalculateMaxTokens calculates the max number of tokens that could be added to a text.

func CountTokens

func CountTokens(model, text string) int

CountTokens gets the number of tokens the text contains.

func GetModelContextSize

func GetModelContextSize(model string) int

ModelContextSize gets the max number of tokens for a language model. If the model name isn't recognized the default value 2048 is returned.

Types

type CallOption

type CallOption func(*CallOptions)

CallOption is a function that configures a CallOptions.

func WithFrequencyPenalty

func WithFrequencyPenalty(frequencyPenalty float64) CallOption

WithFrequencyPenalty will add an option to set the frequency penalty for sampling.

func WithFunctionCallBehavior

func WithFunctionCallBehavior(behavior FunctionCallBehavior) CallOption

WithFunctionCallBehavior will add an option to set the behavior to use when calling functions.

func WithFunctions

func WithFunctions(functions []FunctionDefinition) CallOption

WithFunctions will add an option to set the functions to include in the request.

func WithMaxLength

func WithMaxLength(maxLength int) CallOption

WithMaxLength will add an option to set the maximum length of the generated text.

func WithMaxTokens

func WithMaxTokens(maxTokens int) CallOption

WithMaxTokens is an option for LLM.Call.

func WithMinLength

func WithMinLength(minLength int) CallOption

WithMinLength will add an option to set the minimum length of the generated text.

func WithModel

func WithModel(model string) CallOption

WithModel is an option for LLM.Call.

func WithN

func WithN(n int) CallOption

WithN will add an option to set how many chat completion choices to generate for each input message.

func WithOptions

func WithOptions(options CallOptions) CallOption

WithOptions is an option for LLM.Call.

func WithPresencePenalty

func WithPresencePenalty(presencePenalty float64) CallOption

WithPresencePenalty will add an option to set the presence penalty for sampling.

func WithRepetitionPenalty

func WithRepetitionPenalty(repetitionPenalty float64) CallOption

WithRepetitionPenalty will add an option to set the repetition penalty for sampling.

func WithSeed

func WithSeed(seed int) CallOption

WithSeed will add an option to use deterministic sampling.

func WithStopWords

func WithStopWords(stopWords []string) CallOption

WithStopWords is an option for LLM.Call.

func WithStreamingFunc

func WithStreamingFunc(streamingFunc func(ctx context.Context, chunk []byte) error) CallOption

WithStreamingFunc is an option for LLM.Call that allows streaming responses.

func WithTemperature

func WithTemperature(temperature float64) CallOption

WithTemperature is an option for LLM.Call.

func WithTopK

func WithTopK(topK int) CallOption

WithTopK will add an option to use top-k sampling.

func WithTopP

func WithTopP(topP float64) CallOption

WithTopP will add an option to use top-p sampling.

type CallOptions

type CallOptions struct {
	// Model is the model to use.
	Model string `json:"model"`
	// MaxTokens is the maximum number of tokens to generate.
	MaxTokens int `json:"max_tokens"`
	// Temperature is the temperature for sampling, between 0 and 1.
	Temperature float64 `json:"temperature"`
	// StopWords is a list of words to stop on.
	StopWords []string `json:"stop_words"`
	// StreamingFunc is a function to be called for each chunk of a streaming response.
	// Return an error to stop streaming early.
	StreamingFunc func(ctx context.Context, chunk []byte) error
	// TopK is the number of tokens to consider for top-k sampling.
	TopK int `json:"top_k"`
	// TopP is the cumulative probability for top-p sampling.
	TopP float64 `json:"top_p"`
	// Seed is a seed for deterministic sampling.
	Seed int `json:"seed"`
	// MinLength is the minimum length of the generated text.
	MinLength int `json:"min_length"`
	// MaxLength is the maximum length of the generated text.
	MaxLength int `json:"max_length"`
	// N is how many chat completion choices to generate for each input message.
	N int `json:"n"`
	// RepetitionPenalty is the repetition penalty for sampling.
	RepetitionPenalty float64 `json:"repetition_penalty"`
	// FrequencyPenalty is the frequency penalty for sampling.
	FrequencyPenalty float64 `json:"frequency_penalty"`
	// PresencePenalty is the presence penalty for sampling.
	PresencePenalty float64 `json:"presence_penalty"`

	// Function defitions to include in the request.
	Functions []FunctionDefinition `json:"functions"`
	// FunctionCallBehavior is the behavior to use when calling functions.
	//
	// If a specific function should be invoked, use the format:
	// `{"name": "my_function"}`
	FunctionCallBehavior FunctionCallBehavior `json:"function_call"`
}

CallOptions is a set of options for LLM.Call.

type ChatLLM

type ChatLLM interface {
	Call(ctx context.Context, messages []schema.ChatMessage, options ...CallOption) (*schema.AIChatMessage, error)
	Generate(ctx context.Context, messages [][]schema.ChatMessage, options ...CallOption) ([]*Generation, error)
}

ChatLLM is a langchaingo LLM that can be used for chatting.

type FunctionCallBehavior

type FunctionCallBehavior string

FunctionCallBehavior is the behavior to use when calling functions.

const (
	// FunctionCallBehaviorNone will not call any functions.
	FunctionCallBehaviorNone FunctionCallBehavior = "none"
	// FunctionCallBehaviorAuto will call functions automatically.
	FunctionCallBehaviorAuto FunctionCallBehavior = "auto"
)

type FunctionDefinition

type FunctionDefinition struct {
	// Name is the name of the function.
	Name string `json:"name"`
	// Description is a description of the function.
	Description string `json:"description"`
	// Parameters is a list of parameters for the function.
	Parameters any `json:"parameters"`
}

FunctionDefinition is a definition of a function that can be called by the model.

type Generation

type Generation struct {
	// Text is the generated text.
	Text string `json:"text"`
	// Message stores the potentially generated message.
	Message *schema.AIChatMessage `json:"message"`
	// GenerationInfo is the generation info. This can contain vendor-specific information.
	GenerationInfo map[string]any `json:"generation_info"`
	// StopReason is the reason the generation stopped.
	StopReason string `json:"stop_reason"`
}

Generation is a single generation from a langchaingo LLM.

type LLM

type LLM interface {
	Call(ctx context.Context, prompt string, options ...CallOption) (string, error)
	Generate(ctx context.Context, prompts []string, options ...CallOption) ([]*Generation, error)
}

LLM is a langchaingo Large Language Model.

type LLMResult

type LLMResult struct {
	Generations [][]*Generation
	LLMOutput   map[string]any
}

LLMResult is the class that contains all relevant information for an LLM Result.

func GenerateChatPrompt

func GenerateChatPrompt(ctx context.Context, l ChatLLM, promptValues []schema.PromptValue, options ...CallOption) (LLMResult, error)

func GeneratePrompt

func GeneratePrompt(ctx context.Context, l LLM, promptValues []schema.PromptValue, options ...CallOption) (LLMResult, error)

type LanguageModel

type LanguageModel interface {
	// Take in a list of prompt values and return an LLMResult.
	GeneratePrompt(ctx context.Context, prompts []schema.PromptValue, options ...CallOption) (LLMResult, error)
	// Get the number of tokens present in the text.
	GetNumTokens(text string) int
}

LanguageModel is the interface all language models must implement.