llms

package
v0.1.21 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2024 License: MIT Imports: 10 Imported by: 34

Documentation

Overview

Package llms provides unified support for interacting with different Language Models (LLMs) from various providers. Designed with an extensible architecture, the package facilitates seamless integration of LLMs with a focus on modularity, encapsulation, and easy configurability.

The package includes the following subpackages for LLM providers: 1. Hugging Face: llms/huggingface/ 2. Local LLM: llms/local/ 3. OpenAI: llms/openai/ 4. Google AI: llms/googleai/ 5. Cohere: llms/cohere/

Each subpackage includes provider-specific LLM implementations and helper files for communication with supported LLM providers. The internal directories within these subpackages contain provider-specific client and API implementations.

The `llms.go` file contains the types and interfaces for interacting with different LLMs.

The `options.go` file provides various options and functions to configure the LLMs.

Index

Constants

This section is empty.

Variables

View Source 
var ErrUnexpectedChatMessageType = errors.New("unexpected chat message type")

ErrUnexpectedChatMessageType is returned when a chat message is of an unexpected type.

Functions

func CalculateMaxTokens 

func CalculateMaxTokens(model, text string) int

CalculateMaxTokens calculates the max number of tokens that could be added to a text.

func CountTokens 

func CountTokens(model, text string) int

CountTokens gets the number of tokens the text contains.

func GenerateFromSinglePrompt added in v0.1.13 

func GenerateFromSinglePrompt(ctx context.Context, llm Model, prompt string, options ...CallOption) (string, error)

GenerateFromSinglePrompt is a convenience function for calling an LLM with a single string prompt, expecting a single string response. It's useful for simple, string-only interactions and provides a slightly more ergonomic API than the more general llms.Model.GenerateContent.

func GetBufferString 

func GetBufferString(messages []ChatMessage, humanPrefix string, aiPrefix string) (string, error)

GetBufferString gets the buffer string of messages.

func GetModelContextSize 

func GetModelContextSize(model string) int

GetModelContextSize gets the max number of tokens for a language model. If the model name isn't recognized the default value 2048 is returned.

func ShowMessageContents 

func ShowMessageContents(w io.Writer, msgs []MessageContent)

ShowMessageContents is a debugging helper for MessageContent.

Types

type AIChatMessage 

type AIChatMessage struct {
	// Content is the content of the message.
	Content string `json:"content,omitempty"`

	// FunctionCall represents the model choosing to call a function.
	FunctionCall *FunctionCall `json:"function_call,omitempty"`

	// ToolCalls represents the model choosing to call tools.
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

AIChatMessage is a message sent by an AI.

func (AIChatMessage) GetContent 

func (m AIChatMessage) GetContent() string

func (AIChatMessage) GetFunctionCall 

func (m AIChatMessage) GetFunctionCall() *FunctionCall

func (AIChatMessage) GetType 

func (m AIChatMessage) GetType() ChatMessageType

type BinaryContent 

type BinaryContent struct {
	MIMEType string
	Data     []byte
}

BinaryContent is content holding some binary data with a MIME type.

func BinaryPart 

func BinaryPart(mime string, data []byte) BinaryContent

BinaryPart creates a new BinaryContent from the given MIME type (e.g. "image/png" and binary data).

func (BinaryContent) MarshalJSON 

func (bc BinaryContent) MarshalJSON() ([]byte, error)

func (BinaryContent) String 

func (bc BinaryContent) String() string

func (*BinaryContent) UnmarshalJSON 

func (bc *BinaryContent) UnmarshalJSON(data []byte) error

type CallOption 

type CallOption func(*CallOptions)

CallOption is a function that configures a CallOptions.

func WithCandidateCount added in v0.1.13 

func WithCandidateCount(c int) CallOption

WithCandidateCount specifies the number of response candidates to generate.

func WithFrequencyPenalty 

func WithFrequencyPenalty(frequencyPenalty float64) CallOption

WithFrequencyPenalty will add an option to set the frequency penalty for sampling.

func WithFunctionCallBehavior 

func WithFunctionCallBehavior(behavior FunctionCallBehavior) CallOption

WithFunctionCallBehavior will add an option to set the behavior to use when calling functions. Deprecated: Use WithToolChoice instead.

func WithFunctions 

func WithFunctions(functions []FunctionDefinition) CallOption

WithFunctions will add an option to set the functions to include in the request. Deprecated: Use WithTools instead.

func WithJSONMode added in v0.1.13 

func WithJSONMode() CallOption

WithJSONMode will add an option to set the response format to JSON. This is useful for models that return structured data.

func WithMaxLength 

func WithMaxLength(maxLength int) CallOption

WithMaxLength will add an option to set the maximum length of the generated text.

func WithMaxTokens 

func WithMaxTokens(maxTokens int) CallOption

WithMaxTokens specifies the max number of tokens to generate.

func WithMetadata added in v0.1.13 

func WithMetadata(metadata map[string]interface{}) CallOption

WithMetadata will add an option to set metadata to include in the request. The meaning of this field is specific to the backend in use.

func WithMinLength 

func WithMinLength(minLength int) CallOption

WithMinLength will add an option to set the minimum length of the generated text.

func WithModel 

func WithModel(model string) CallOption

WithModel specifies which model name to use.

func WithN 

func WithN(n int) CallOption

WithN will add an option to set how many chat completion choices to generate for each input message.

func WithOptions 

func WithOptions(options CallOptions) CallOption

WithOptions specifies options.

func WithPresencePenalty 

func WithPresencePenalty(presencePenalty float64) CallOption

WithPresencePenalty will add an option to set the presence penalty for sampling.

func WithRepetitionPenalty 

func WithRepetitionPenalty(repetitionPenalty float64) CallOption

WithRepetitionPenalty will add an option to set the repetition penalty for sampling.

func WithResponseMIMEType added in v0.1.21 

func WithResponseMIMEType(responseMIMEType string) CallOption

WithResponseMIMEType will add an option to set the ResponseMIMEType Currently only supported by googleai llms.

func WithSeed 

func WithSeed(seed int) CallOption

WithSeed will add an option to use deterministic sampling.

func WithStopWords 

func WithStopWords(stopWords []string) CallOption

WithStopWords specifies a list of words to stop generation on.

func WithStreamingFunc 

func WithStreamingFunc(streamingFunc func(ctx context.Context, chunk []byte) error) CallOption

WithStreamingFunc specifies the streaming function to use.

func WithTemperature 

func WithTemperature(temperature float64) CallOption

WithTemperature specifies the model temperature, a hyperparameter that regulates the randomness, or creativity, of the AI's responses.

func WithToolChoice added in v0.1.13 

func WithToolChoice(choice any) CallOption

WithToolChoice will add an option to set the choice of tool to use. It can either be "none", "auto" (the default behavior), or a specific tool as described in the ToolChoice type.

func WithTools added in v0.1.13 

func WithTools(tools []Tool) CallOption

WithTools will add an option to set the tools to use.

func WithTopK 

func WithTopK(topK int) CallOption

WithTopK will add an option to use top-k sampling.

func WithTopP 

func WithTopP(topP float64) CallOption

WithTopP will add an option to use top-p sampling.

type CallOptions 

type CallOptions struct {
	// Model is the model to use.
	Model string `json:"model"`
	// CandidateCount is the number of response candidates to generate.
	CandidateCount int `json:"candidate_count"`
	// MaxTokens is the maximum number of tokens to generate.
	MaxTokens int `json:"max_tokens"`
	// Temperature is the temperature for sampling, between 0 and 1.
	Temperature float64 `json:"temperature"`
	// StopWords is a list of words to stop on.
	StopWords []string `json:"stop_words"`
	// StreamingFunc is a function to be called for each chunk of a streaming response.
	// Return an error to stop streaming early.
	StreamingFunc func(ctx context.Context, chunk []byte) error `json:"-"`
	// TopK is the number of tokens to consider for top-k sampling.
	TopK int `json:"top_k"`
	// TopP is the cumulative probability for top-p sampling.
	TopP float64 `json:"top_p"`
	// Seed is a seed for deterministic sampling.
	Seed int `json:"seed"`
	// MinLength is the minimum length of the generated text.
	MinLength int `json:"min_length"`
	// MaxLength is the maximum length of the generated text.
	MaxLength int `json:"max_length"`
	// N is how many chat completion choices to generate for each input message.
	N int `json:"n"`
	// RepetitionPenalty is the repetition penalty for sampling.
	RepetitionPenalty float64 `json:"repetition_penalty"`
	// FrequencyPenalty is the frequency penalty for sampling.
	FrequencyPenalty float64 `json:"frequency_penalty"`
	// PresencePenalty is the presence penalty for sampling.
	PresencePenalty float64 `json:"presence_penalty"`

	// JSONMode is a flag to enable JSON mode.
	JSONMode bool `json:"json"`

	// Tools is a list of tools to use. Each tool can be a specific tool or a function.
	Tools []Tool `json:"tools,omitempty"`
	// ToolChoice is the choice of tool to use, it can either be "none", "auto" (the default behavior), or a specific tool as described in the ToolChoice type.
	ToolChoice any `json:"tool_choice"`

	// Function defitions to include in the request.
	// Deprecated: Use Tools instead.
	Functions []FunctionDefinition `json:"functions,omitempty"`
	// FunctionCallBehavior is the behavior to use when calling functions.
	//
	// If a specific function should be invoked, use the format:
	// `{"name": "my_function"}`
	// Deprecated: Use ToolChoice instead.
	FunctionCallBehavior FunctionCallBehavior `json:"function_call,omitempty"`

	// Metadata is a map of metadata to include in the request.
	// The meaning of this field is specific to the backend in use.
	Metadata map[string]interface{} `json:"metadata,omitempty"`

	// ResponseMIMEType MIME type of the generated candidate text.
	// Supported MIME types are: text/plain: (default) Text output.
	// application/json: JSON response in the response candidates.
	ResponseMIMEType string `json:"response_mime_type,omitempty"`
}

CallOptions is a set of options for calling models. Not all models support all options.

type ChatMessage 

type ChatMessage interface {
	// GetType gets the type of the message.
	GetType() ChatMessageType
	// GetContent gets the content of the message.
	GetContent() string
}

ChatMessage represents a message in a chat.

type ChatMessageModel 

type ChatMessageModel struct {
	Type string               `bson:"type" json:"type"`
	Data ChatMessageModelData `bson:"data" json:"data"`
}

func ConvertChatMessageToModel 

func ConvertChatMessageToModel(m ChatMessage) ChatMessageModel

ConvertChatMessageToModel Convert a ChatMessage to a ChatMessageModel.

func (ChatMessageModel) ToChatMessage 

func (c ChatMessageModel) ToChatMessage() ChatMessage

type ChatMessageModelData 

type ChatMessageModelData struct {
	Content string `bson:"content" json:"content"`
	Type    string `bson:"type"    json:"type"`
}

type ChatMessageType 

type ChatMessageType string

ChatMessageType is the type of chat message. 

const (
	// ChatMessageTypeAI is a message sent by an AI.
	ChatMessageTypeAI ChatMessageType = "ai"
	// ChatMessageTypeHuman is a message sent by a human.
	ChatMessageTypeHuman ChatMessageType = "human"
	// ChatMessageTypeSystem is a message sent by the system.
	ChatMessageTypeSystem ChatMessageType = "system"
	// ChatMessageTypeGeneric is a message sent by a generic user.
	ChatMessageTypeGeneric ChatMessageType = "generic"
	// ChatMessageTypeFunction is a message sent by a function.
	ChatMessageTypeFunction ChatMessageType = "function"
	// ChatMessageTypeTool is a message sent by a tool.
	ChatMessageTypeTool ChatMessageType = "tool"
)

type ContentChoice 

type ContentChoice struct {
	// Content is the textual content of a response
	Content string

	// StopReason is the reason the model stopped generating output.
	StopReason string

	// GenerationInfo is arbitrary information the model adds to the response.
	GenerationInfo map[string]any

	// FuncCall is non-nil when the model asks to invoke a function/tool.
	// If a model invokes more than one function/tool, this field will only
	// contain the first one.
	FuncCall *FunctionCall

	// ToolCalls is a list of tool calls the model asks to invoke.
	ToolCalls []ToolCall
}

ContentChoice is one of the response choices returned by GenerateContent calls.

type ContentPart 

type ContentPart interface {
	// contains filtered or unexported methods
}

ContentPart is an interface all parts of content have to implement.

type ContentResponse 

type ContentResponse struct {
	Choices []*ContentChoice
}

ContentResponse is the response returned by a GenerateContent call. It can potentially return multiple content choices.

type FunctionCall 

type FunctionCall struct {
	// The name of the function to call.
	Name string `json:"name"`
	// The arguments to pass to the function, as a JSON string.
	Arguments string `json:"arguments"`
}

FunctionCall is the name and arguments of a function call.

type FunctionCallBehavior 

type FunctionCallBehavior string

FunctionCallBehavior is the behavior to use when calling functions. 

const (
	// FunctionCallBehaviorNone will not call any functions.
	FunctionCallBehaviorNone FunctionCallBehavior = "none"
	// FunctionCallBehaviorAuto will call functions automatically.
	FunctionCallBehaviorAuto FunctionCallBehavior = "auto"
)

type FunctionChatMessage 

type FunctionChatMessage struct {
	// Name is the name of the function.
	Name string `json:"name"`
	// Content is the content of the function message.
	Content string `json:"content"`
}

FunctionChatMessage is a chat message representing the result of a function call. Deprecated: Use ToolChatMessage instead.

func (FunctionChatMessage) GetContent 

func (m FunctionChatMessage) GetContent() string

func (FunctionChatMessage) GetName 

func (m FunctionChatMessage) GetName() string

func (FunctionChatMessage) GetType 

func (m FunctionChatMessage) GetType() ChatMessageType

type FunctionDefinition 

type FunctionDefinition struct {
	// Name is the name of the function.
	Name string `json:"name"`
	// Description is a description of the function.
	Description string `json:"description"`
	// Parameters is a list of parameters for the function.
	Parameters any `json:"parameters,omitempty"`
	// Strict is a flag to indicate if the function should be called strictly. Only used for openai llm structured output.
	Strict bool `json:"strict,omitempty"`
}

FunctionDefinition is a definition of a function that can be called by the model.

type FunctionReference added in v0.1.10 

type FunctionReference struct {
	// Name is the name of the function.
	Name string `json:"name"`
}

FunctionReference is a reference to a function.

type GenericChatMessage 

type GenericChatMessage struct {
	Content string
	Role    string
	Name    string
}

GenericChatMessage is a chat message with an arbitrary speaker.

func (GenericChatMessage) GetContent 

func (m GenericChatMessage) GetContent() string

func (GenericChatMessage) GetName 

func (m GenericChatMessage) GetName() string

func (GenericChatMessage) GetType 

func (m GenericChatMessage) GetType() ChatMessageType

type HumanChatMessage 

type HumanChatMessage struct {
	Content string
}

HumanChatMessage is a message sent by a human.

func (HumanChatMessage) GetContent 

func (m HumanChatMessage) GetContent() string

func (HumanChatMessage) GetType 

func (m HumanChatMessage) GetType() ChatMessageType

type ImageURLContent 

type ImageURLContent struct {
	URL    string `json:"url"`
	Detail string `json:"detail,omitempty"` // Detail is the detail of the image, e.g. "low", "high".
}

ImageURLContent is content with an URL pointing to an image.

func ImageURLPart 

func ImageURLPart(url string) ImageURLContent

ImageURLPart creates a new ImageURLContent from the given URL.

func ImageURLWithDetailPart 

func ImageURLWithDetailPart(url string, detail string) ImageURLContent

ImageURLWithDetailPart creates a new ImageURLContent from the given URL and detail.

func (ImageURLContent) MarshalJSON 

func (iuc ImageURLContent) MarshalJSON() ([]byte, error)

func (ImageURLContent) String 

func (iuc ImageURLContent) String() string

func (*ImageURLContent) UnmarshalJSON 

func (iuc *ImageURLContent) UnmarshalJSON(data []byte) error

type LLM deprecated 

type LLM = Model

LLM is an alias for model, for backwards compatibility.

Deprecated: This alias may be removed in the future; please use Model instead.

type MessageContent 

type MessageContent struct {
	Role  ChatMessageType
	Parts []ContentPart
}

MessageContent is the content of a message sent to a LLM. It has a role and a sequence of parts. For example, it can represent one message in a chat session sent by the user, in which case Role will be ChatMessageTypeHuman and Parts will be the sequence of items sent in this specific message.

func TextParts 

func TextParts(role ChatMessageType, parts ...string) MessageContent

TextParts is a helper function to create a MessageContent with a role and a list of text parts.

func (MessageContent) MarshalJSON 

func (mc MessageContent) MarshalJSON() ([]byte, error)

func (*MessageContent) UnmarshalJSON 

func (mc *MessageContent) UnmarshalJSON(data []byte) error

type Model added in v0.1.13 

type Model interface {
	// GenerateContent asks the model to generate content from a sequence of
	// messages. It's the most general interface for multi-modal LLMs that support
	// chat-like interactions.
	GenerateContent(ctx context.Context, messages []MessageContent, options ...CallOption) (*ContentResponse, error)

	// Call is a simplified interface for a text-only Model, generating a single
	// string response from a single string prompt.
	//
	// Deprecated: this method is retained for backwards compatibility. Use the
	// more general [GenerateContent] instead. You can also use
	// the [GenerateFromSinglePrompt] function which provides a similar capability
	// to Call and is built on top of the new interface.
	Call(ctx context.Context, prompt string, options ...CallOption) (string, error)
}

Model is an interface multi-modal models implement.

type Named 

type Named interface {
	GetName() string
}

Named is an interface for objects that have a name.

type PromptValue 

type PromptValue interface {
	String() string
	Messages() []ChatMessage
}

PromptValue is the interface that all prompt values must implement.

type SystemChatMessage 

type SystemChatMessage struct {
	Content string
}

SystemChatMessage is a chat message representing information that should be instructions to the AI system.

func (SystemChatMessage) GetContent 

func (m SystemChatMessage) GetContent() string

func (SystemChatMessage) GetType 

func (m SystemChatMessage) GetType() ChatMessageType

type TextContent 

type TextContent struct {
	Text string
}

TextContent is content with some text.

func TextPart 

func TextPart(s string) TextContent

TextPart creates TextContent from a given string.

func (TextContent) MarshalJSON 

func (tc TextContent) MarshalJSON() ([]byte, error)

func (TextContent) String 

func (tc TextContent) String() string

func (*TextContent) UnmarshalJSON 

func (tc *TextContent) UnmarshalJSON(data []byte) error

type Tool added in v0.1.10 

type Tool struct {
	// Type is the type of the tool.
	Type string `json:"type"`
	// Function is the function to call.
	Function *FunctionDefinition `json:"function,omitempty"`
}

Tool is a tool that can be used by the model.

type ToolCall 

type ToolCall struct {
	// ID is the unique identifier of the tool call.
	ID string `json:"id"`
	// Type is the type of the tool call. Typically, this would be "function".
	Type string `json:"type"`
	// FunctionCall is the function call to be executed.
	FunctionCall *FunctionCall `json:"function,omitempty"`
}

ToolCall is a call to a tool (as requested by the model) that should be executed.

func (ToolCall) MarshalJSON 

func (tc ToolCall) MarshalJSON() ([]byte, error)

func (*ToolCall) UnmarshalJSON 

func (tc *ToolCall) UnmarshalJSON(data []byte) error

type ToolCallResponse 

type ToolCallResponse struct {
	// ToolCallID is the ID of the tool call this response is for.
	ToolCallID string `json:"tool_call_id"`
	// Name is the name of the tool that was called.
	Name string `json:"name"`
	// Content is the textual content of the response.
	Content string `json:"content"`
}

ToolCallResponse is the response returned by a tool call.

func (ToolCallResponse) MarshalJSON 

func (tc ToolCallResponse) MarshalJSON() ([]byte, error)

func (*ToolCallResponse) UnmarshalJSON 

func (tc *ToolCallResponse) UnmarshalJSON(data []byte) error

type ToolChatMessage 

type ToolChatMessage struct {
	// ID is the ID of the tool call.
	ID string `json:"tool_call_id"`
	// Content is the content of the tool message.
	Content string `json:"content"`
}

ToolChatMessage is a chat message representing the result of a tool call.

func (ToolChatMessage) GetContent 

func (m ToolChatMessage) GetContent() string

func (ToolChatMessage) GetID 

func (m ToolChatMessage) GetID() string

func (ToolChatMessage) GetType 

func (m ToolChatMessage) GetType() ChatMessageType

type ToolChoice added in v0.1.10 

type ToolChoice struct {
	// Type is the type of the tool.
	Type string `json:"type"`
	// Function is the function to call (if the tool is a function).
	Function *FunctionReference `json:"function,omitempty"`
}

ToolChoice is a specific tool to use.

Source Files

View all Source files

Directories

Path Synopsis
internal/anthropicclient
internal/bedrockclient
Package cache provides a generic wrapper that adds caching to a `llms.Model`.
 Package cache provides a generic wrapper that adds caching to a `llms.Model`.
inmemory
internal/cloudflareclient
internal/cohereclient
Package ernie wrapper around the Baidu Large Language Model Platform APIs.
 Package ernie wrapper around the Baidu Large Language Model Platform APIs.
internal/ernieclient
package googleai implements a langchaingo provider for Google AI LLMs.
 package googleai implements a langchaingo provider for Google AI LLMs.
internal/cmd
Code generator for vertex.go from googleai.go nolint
 Code generator for vertex.go from googleai.go nolint
internal/palmclient
palm
package palm implements a langchaingo provider for Google Vertex AI legacy PaLM models.
 package palm implements a langchaingo provider for Google Vertex AI legacy PaLM models.
vertex
package vertex implements a langchaingo provider for Google Vertex AI LLMs, including the new Gemini models.
 package vertex implements a langchaingo provider for Google Vertex AI LLMs, including the new Gemini models.
internal/huggingfaceclient
internal/llamafileclient
internal/localclient
Package localclient provides a client for local LLMs.
 Package localclient provides a client for local LLMs.
internal/maritacaclient
internal/ollamaclient
internal/openaiclient

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.