Documentation ¶
Overview ¶
Package llms provides unified support for interacting with different Language Models (LLMs) from various providers. Designed with an extensible architecture, the package facilitates seamless integration of LLMs with a focus on modularity, encapsulation, and easy configurability.
The package includes the following subpackages for LLM providers: 1. Hugging Face: llms/huggingface/ 2. Local LLM: llms/local/ 3. OpenAI: llms/openai/ 4. Google AI: llms/googleai/ 5. Cohere: llms/cohere/
Each subpackage includes provider-specific LLM implementations and helper files for communication with supported LLM providers. The internal directories within these subpackages contain provider-specific client and API implementations.
The `llms.go` file contains the types and interfaces for interacting with different LLMs.
The `options.go` file provides various options and functions to configure the LLMs.
Index ¶
- Variables
- func CalculateMaxTokens(model, text string) int
- func CountTokens(model, text string) int
- func GenerateFromSinglePrompt(ctx context.Context, llm Model, prompt string, options ...CallOption) (string, error)
- func GetBufferString(messages []ChatMessage, humanPrefix string, aiPrefix string) (string, error)
- func GetModelContextSize(model string) int
- func ShowMessageContents(w io.Writer, msgs []MessageContent)
- type AIChatMessage
- type BinaryContent
- type CallOption
- func WithCandidateCount(c int) CallOption
- func WithFrequencyPenalty(frequencyPenalty float64) CallOption
- func WithFunctionCallBehavior(behavior FunctionCallBehavior) CallOption
- func WithFunctions(functions []FunctionDefinition) CallOption
- func WithJSONMode() CallOption
- func WithMaxLength(maxLength int) CallOption
- func WithMaxTokens(maxTokens int) CallOption
- func WithMetadata(metadata map[string]interface{}) CallOption
- func WithMinLength(minLength int) CallOption
- func WithModel(model string) CallOption
- func WithN(n int) CallOption
- func WithOptions(options CallOptions) CallOption
- func WithPresencePenalty(presencePenalty float64) CallOption
- func WithRepetitionPenalty(repetitionPenalty float64) CallOption
- func WithResponseMIMEType(responseMIMEType string) CallOption
- func WithSeed(seed int) CallOption
- func WithStopWords(stopWords []string) CallOption
- func WithStreamingFunc(streamingFunc func(ctx context.Context, chunk []byte) error) CallOption
- func WithTemperature(temperature float64) CallOption
- func WithToolChoice(choice any) CallOption
- func WithTools(tools []Tool) CallOption
- func WithTopK(topK int) CallOption
- func WithTopP(topP float64) CallOption
- type CallOptions
- type ChatMessage
- type ChatMessageModel
- type ChatMessageModelData
- type ChatMessageType
- type ContentChoice
- type ContentPart
- type ContentResponse
- type FunctionCall
- type FunctionCallBehavior
- type FunctionChatMessage
- type FunctionDefinition
- type FunctionReference
- type GenericChatMessage
- type HumanChatMessage
- type ImageURLContent
- type LLMdeprecated
- type MessageContent
- type Model
- type Named
- type PromptValue
- type SystemChatMessage
- type TextContent
- type Tool
- type ToolCall
- type ToolCallResponse
- type ToolChatMessage
- type ToolChoice
Constants ¶
This section is empty.
Variables ¶
var ErrUnexpectedChatMessageType = errors.New("unexpected chat message type")
ErrUnexpectedChatMessageType is returned when a chat message is of an unexpected type.
Functions ¶
func CalculateMaxTokens ¶
CalculateMaxTokens calculates the max number of tokens that could be added to a text.
func CountTokens ¶
CountTokens gets the number of tokens the text contains.
func GenerateFromSinglePrompt ¶ added in v0.1.4
func GenerateFromSinglePrompt(ctx context.Context, llm Model, prompt string, options ...CallOption) (string, error)
GenerateFromSinglePrompt is a convenience function for calling an LLM with a single string prompt, expecting a single string response. It's useful for simple, string-only interactions and provides a slightly more ergonomic API than the more general llms.Model.GenerateContent.
func GetBufferString ¶ added in v0.1.9
func GetBufferString(messages []ChatMessage, humanPrefix string, aiPrefix string) (string, error)
GetBufferString gets the buffer string of messages.
func GetModelContextSize ¶
GetModelContextSize gets the max number of tokens for a language model. If the model name isn't recognized the default value 2048 is returned.
func ShowMessageContents ¶ added in v0.1.10
func ShowMessageContents(w io.Writer, msgs []MessageContent)
ShowMessageContents is a debugging helper for MessageContent.
Types ¶
type AIChatMessage ¶ added in v0.1.9
type AIChatMessage struct { // Content is the content of the message. Content string `json:"content,omitempty"` // FunctionCall represents the model choosing to call a function. FunctionCall *FunctionCall `json:"function_call,omitempty"` // ToolCalls represents the model choosing to call tools. ToolCalls []ToolCall `json:"tool_calls,omitempty"` }
AIChatMessage is a message sent by an AI.
func (AIChatMessage) GetContent ¶ added in v0.1.9
func (m AIChatMessage) GetContent() string
func (AIChatMessage) GetFunctionCall ¶ added in v0.1.9
func (m AIChatMessage) GetFunctionCall() *FunctionCall
func (AIChatMessage) GetType ¶ added in v0.1.9
func (m AIChatMessage) GetType() ChatMessageType
type BinaryContent ¶ added in v0.1.3
BinaryContent is content holding some binary data with a MIME type.
func BinaryPart ¶ added in v0.1.4
func BinaryPart(mime string, data []byte) BinaryContent
BinaryPart creates a new BinaryContent from the given MIME type (e.g. "image/png" and binary data).
func (BinaryContent) MarshalJSON ¶ added in v0.1.10
func (bc BinaryContent) MarshalJSON() ([]byte, error)
func (BinaryContent) String ¶ added in v0.1.10
func (bc BinaryContent) String() string
func (*BinaryContent) UnmarshalJSON ¶ added in v0.1.11
func (bc *BinaryContent) UnmarshalJSON(data []byte) error
type CallOption ¶
type CallOption func(*CallOptions)
CallOption is a function that configures a CallOptions.
func WithCandidateCount ¶ added in v0.1.4
func WithCandidateCount(c int) CallOption
WithCandidateCount specifies the number of response candidates to generate.
func WithFrequencyPenalty ¶
func WithFrequencyPenalty(frequencyPenalty float64) CallOption
WithFrequencyPenalty will add an option to set the frequency penalty for sampling.
func WithFunctionCallBehavior ¶
func WithFunctionCallBehavior(behavior FunctionCallBehavior) CallOption
WithFunctionCallBehavior will add an option to set the behavior to use when calling functions. Deprecated: Use WithToolChoice instead.
func WithFunctions ¶
func WithFunctions(functions []FunctionDefinition) CallOption
WithFunctions will add an option to set the functions to include in the request. Deprecated: Use WithTools instead.
func WithJSONMode ¶ added in v0.1.6
func WithJSONMode() CallOption
WithJSONMode will add an option to set the response format to JSON. This is useful for models that return structured data.
func WithMaxLength ¶
func WithMaxLength(maxLength int) CallOption
WithMaxLength will add an option to set the maximum length of the generated text.
func WithMaxTokens ¶
func WithMaxTokens(maxTokens int) CallOption
WithMaxTokens specifies the max number of tokens to generate.
func WithMetadata ¶ added in v0.1.10
func WithMetadata(metadata map[string]interface{}) CallOption
WithMetadata will add an option to set metadata to include in the request. The meaning of this field is specific to the backend in use.
func WithMinLength ¶
func WithMinLength(minLength int) CallOption
WithMinLength will add an option to set the minimum length of the generated text.
func WithModel ¶
func WithModel(model string) CallOption
WithModel specifies which model name to use.
func WithN ¶
func WithN(n int) CallOption
WithN will add an option to set how many chat completion choices to generate for each input message.
func WithPresencePenalty ¶
func WithPresencePenalty(presencePenalty float64) CallOption
WithPresencePenalty will add an option to set the presence penalty for sampling.
func WithRepetitionPenalty ¶
func WithRepetitionPenalty(repetitionPenalty float64) CallOption
WithRepetitionPenalty will add an option to set the repetition penalty for sampling.
func WithResponseMIMEType ¶
func WithResponseMIMEType(responseMIMEType string) CallOption
WithResponseMIMEType will add an option to set the ResponseMIMEType Currently only supported by googleai llms.
func WithSeed ¶
func WithSeed(seed int) CallOption
WithSeed will add an option to use deterministic sampling.
func WithStopWords ¶
func WithStopWords(stopWords []string) CallOption
WithStopWords specifies a list of words to stop generation on.
func WithStreamingFunc ¶
func WithStreamingFunc(streamingFunc func(ctx context.Context, chunk []byte) error) CallOption
WithStreamingFunc specifies the streaming function to use.
func WithTemperature ¶
func WithTemperature(temperature float64) CallOption
WithTemperature specifies the model temperature, a hyperparameter that regulates the randomness, or creativity, of the AI's responses.
func WithToolChoice ¶ added in v0.1.8
func WithToolChoice(choice any) CallOption
WithToolChoice will add an option to set the choice of tool to use. It can either be "none", "auto" (the default behavior), or a specific tool as described in the ToolChoice type.
func WithTools ¶ added in v0.1.8
func WithTools(tools []Tool) CallOption
WithTools will add an option to set the tools to use.
func WithTopK ¶
func WithTopK(topK int) CallOption
WithTopK will add an option to use top-k sampling.
func WithTopP ¶
func WithTopP(topP float64) CallOption
WithTopP will add an option to use top-p sampling.
type CallOptions ¶
type CallOptions struct { // Model is the model to use. Model string `json:"model"` // CandidateCount is the number of response candidates to generate. CandidateCount int `json:"candidate_count"` // MaxTokens is the maximum number of tokens to generate. MaxTokens int `json:"max_tokens"` // Temperature is the temperature for sampling, between 0 and 1. Temperature float64 `json:"temperature"` // StopWords is a list of words to stop on. StopWords []string `json:"stop_words"` // StreamingFunc is a function to be called for each chunk of a streaming response. // Return an error to stop streaming early. StreamingFunc func(ctx context.Context, chunk []byte) error `json:"-"` // TopK is the number of tokens to consider for top-k sampling. TopK int `json:"top_k"` // TopP is the cumulative probability for top-p sampling. TopP float64 `json:"top_p"` // Seed is a seed for deterministic sampling. Seed int `json:"seed"` // MinLength is the minimum length of the generated text. MinLength int `json:"min_length"` // MaxLength is the maximum length of the generated text. MaxLength int `json:"max_length"` // N is how many chat completion choices to generate for each input message. N int `json:"n"` // RepetitionPenalty is the repetition penalty for sampling. RepetitionPenalty float64 `json:"repetition_penalty"` // FrequencyPenalty is the frequency penalty for sampling. FrequencyPenalty float64 `json:"frequency_penalty"` // PresencePenalty is the presence penalty for sampling. PresencePenalty float64 `json:"presence_penalty"` // JSONMode is a flag to enable JSON mode. JSONMode bool `json:"json"` // Tools is a list of tools to use. Each tool can be a specific tool or a function. Tools []Tool `json:"tools,omitempty"` // ToolChoice is the choice of tool to use, it can either be "none", "auto" (the default behavior), or a specific tool as described in the ToolChoice type. ToolChoice any `json:"tool_choice"` // Function defitions to include in the request. // Deprecated: Use Tools instead. Functions []FunctionDefinition `json:"functions,omitempty"` // FunctionCallBehavior is the behavior to use when calling functions. // // If a specific function should be invoked, use the format: // `{"name": "my_function"}` // Deprecated: Use ToolChoice instead. FunctionCallBehavior FunctionCallBehavior `json:"function_call,omitempty"` // Metadata is a map of metadata to include in the request. // The meaning of this field is specific to the backend in use. Metadata map[string]interface{} `json:"metadata,omitempty"` // ResponseMIMEType MIME type of the generated candidate text. // Supported MIME types are: text/plain: (default) Text output. // application/json: JSON response in the response candidates. ResponseMIMEType string `json:"response_mime_type,omitempty"` }
CallOptions is a set of options for calling models. Not all models support all options.
type ChatMessage ¶ added in v0.1.9
type ChatMessage interface { // GetType gets the type of the message. GetType() ChatMessageType // GetContent gets the content of the message. GetContent() string }
ChatMessage represents a message in a chat.
type ChatMessageModel ¶ added in v0.1.10
type ChatMessageModel struct { Type string `bson:"type" json:"type"` Data ChatMessageModelData `bson:"data" json:"data"` }
func ConvertChatMessageToModel ¶ added in v0.1.10
func ConvertChatMessageToModel(m ChatMessage) ChatMessageModel
ConvertChatMessageToModel Convert a ChatMessage to a ChatMessageModel.
func (ChatMessageModel) ToChatMessage ¶ added in v0.1.10
func (c ChatMessageModel) ToChatMessage() ChatMessage
type ChatMessageModelData ¶ added in v0.1.10
type ChatMessageType ¶ added in v0.1.9
type ChatMessageType string
ChatMessageType is the type of chat message.
const ( // ChatMessageTypeAI is a message sent by an AI. ChatMessageTypeAI ChatMessageType = "ai" // ChatMessageTypeHuman is a message sent by a human. ChatMessageTypeHuman ChatMessageType = "human" // ChatMessageTypeSystem is a message sent by the system. ChatMessageTypeSystem ChatMessageType = "system" // ChatMessageTypeGeneric is a message sent by a generic user. ChatMessageTypeGeneric ChatMessageType = "generic" // ChatMessageTypeFunction is a message sent by a function. ChatMessageTypeFunction ChatMessageType = "function" // ChatMessageTypeTool is a message sent by a tool. ChatMessageTypeTool ChatMessageType = "tool" )
type ContentChoice ¶ added in v0.1.3
type ContentChoice struct { // Content is the textual content of a response Content string // StopReason is the reason the model stopped generating output. StopReason string // GenerationInfo is arbitrary information the model adds to the response. GenerationInfo map[string]any // FuncCall is non-nil when the model asks to invoke a function/tool. // If a model invokes more than one function/tool, this field will only // contain the first one. FuncCall *FunctionCall // ToolCalls is a list of tool calls the model asks to invoke. ToolCalls []ToolCall }
ContentChoice is one of the response choices returned by GenerateContent calls.
type ContentPart ¶ added in v0.1.3
type ContentPart interface {
// contains filtered or unexported methods
}
ContentPart is an interface all parts of content have to implement.
type ContentResponse ¶ added in v0.1.3
type ContentResponse struct {
Choices []*ContentChoice
}
ContentResponse is the response returned by a GenerateContent call. It can potentially return multiple content choices.
type FunctionCall ¶ added in v0.1.9
type FunctionCall struct { // The name of the function to call. Name string `json:"name"` // The arguments to pass to the function, as a JSON string. Arguments string `json:"arguments"` }
FunctionCall is the name and arguments of a function call.
type FunctionCallBehavior ¶
type FunctionCallBehavior string
FunctionCallBehavior is the behavior to use when calling functions.
const ( // FunctionCallBehaviorNone will not call any functions. FunctionCallBehaviorNone FunctionCallBehavior = "none" // FunctionCallBehaviorAuto will call functions automatically. FunctionCallBehaviorAuto FunctionCallBehavior = "auto" )
type FunctionChatMessage ¶ added in v0.1.9
type FunctionChatMessage struct { // Name is the name of the function. Name string `json:"name"` // Content is the content of the function message. Content string `json:"content"` }
FunctionChatMessage is a chat message representing the result of a function call. Deprecated: Use ToolChatMessage instead.
func (FunctionChatMessage) GetContent ¶ added in v0.1.9
func (m FunctionChatMessage) GetContent() string
func (FunctionChatMessage) GetName ¶ added in v0.1.9
func (m FunctionChatMessage) GetName() string
func (FunctionChatMessage) GetType ¶ added in v0.1.9
func (m FunctionChatMessage) GetType() ChatMessageType
type FunctionDefinition ¶
type FunctionDefinition struct { // Name is the name of the function. Name string `json:"name"` // Description is a description of the function. Description string `json:"description"` // Parameters is a list of parameters for the function. Parameters any `json:"parameters,omitempty"` // Strict is a flag to indicate if the function should be called strictly. Only used for openai llm structured output. Strict bool `json:"strict,omitempty"` }
FunctionDefinition is a definition of a function that can be called by the model.
type FunctionReference ¶ added in v0.1.8
type FunctionReference struct { // Name is the name of the function. Name string `json:"name"` }
FunctionReference is a reference to a function.
type GenericChatMessage ¶ added in v0.1.9
GenericChatMessage is a chat message with an arbitrary speaker.
func (GenericChatMessage) GetContent ¶ added in v0.1.9
func (m GenericChatMessage) GetContent() string
func (GenericChatMessage) GetName ¶ added in v0.1.9
func (m GenericChatMessage) GetName() string
func (GenericChatMessage) GetType ¶ added in v0.1.9
func (m GenericChatMessage) GetType() ChatMessageType
type HumanChatMessage ¶ added in v0.1.9
type HumanChatMessage struct {
Content string
}
HumanChatMessage is a message sent by a human.
func (HumanChatMessage) GetContent ¶ added in v0.1.9
func (m HumanChatMessage) GetContent() string
func (HumanChatMessage) GetType ¶ added in v0.1.9
func (m HumanChatMessage) GetType() ChatMessageType
type ImageURLContent ¶ added in v0.1.3
type ImageURLContent struct { URL string `json:"url"` Detail string `json:"detail,omitempty"` // Detail is the detail of the image, e.g. "low", "high". }
ImageURLContent is content with an URL pointing to an image.
func ImageURLPart ¶ added in v0.1.4
func ImageURLPart(url string) ImageURLContent
ImageURLPart creates a new ImageURLContent from the given URL.
func ImageURLWithDetailPart ¶ added in v0.1.11
func ImageURLWithDetailPart(url string, detail string) ImageURLContent
ImageURLWithDetailPart creates a new ImageURLContent from the given URL and detail.
func (ImageURLContent) MarshalJSON ¶ added in v0.1.3
func (iuc ImageURLContent) MarshalJSON() ([]byte, error)
func (ImageURLContent) String ¶ added in v0.1.10
func (iuc ImageURLContent) String() string
func (*ImageURLContent) UnmarshalJSON ¶ added in v0.1.11
func (iuc *ImageURLContent) UnmarshalJSON(data []byte) error
type MessageContent ¶ added in v0.1.4
type MessageContent struct { Role ChatMessageType Parts []ContentPart }
MessageContent is the content of a message sent to a LLM. It has a role and a sequence of parts. For example, it can represent one message in a chat session sent by the user, in which case Role will be ChatMessageTypeHuman and Parts will be the sequence of items sent in this specific message.
func TextParts ¶ added in v0.1.4
func TextParts(role ChatMessageType, parts ...string) MessageContent
TextParts is a helper function to create a MessageContent with a role and a list of text parts.
func (MessageContent) MarshalJSON ¶ added in v0.1.11
func (mc MessageContent) MarshalJSON() ([]byte, error)
func (*MessageContent) UnmarshalJSON ¶ added in v0.1.11
func (mc *MessageContent) UnmarshalJSON(data []byte) error
type Model ¶ added in v0.1.3
type Model interface { // GenerateContent asks the model to generate content from a sequence of // messages. It's the most general interface for multi-modal LLMs that support // chat-like interactions. GenerateContent(ctx context.Context, messages []MessageContent, options ...CallOption) (*ContentResponse, error) // Call is a simplified interface for a text-only Model, generating a single // string response from a single string prompt. // // Deprecated: this method is retained for backwards compatibility. Use the // more general [GenerateContent] instead. You can also use // the [GenerateFromSinglePrompt] function which provides a similar capability // to Call and is built on top of the new interface. Call(ctx context.Context, prompt string, options ...CallOption) (string, error) }
Model is an interface multi-modal models implement.
type Named ¶ added in v0.1.9
type Named interface {
GetName() string
}
Named is an interface for objects that have a name.
type PromptValue ¶ added in v0.1.9
type PromptValue interface { String() string Messages() []ChatMessage }
PromptValue is the interface that all prompt values must implement.
type SystemChatMessage ¶ added in v0.1.9
type SystemChatMessage struct {
Content string
}
SystemChatMessage is a chat message representing information that should be instructions to the AI system.
func (SystemChatMessage) GetContent ¶ added in v0.1.9
func (m SystemChatMessage) GetContent() string
func (SystemChatMessage) GetType ¶ added in v0.1.9
func (m SystemChatMessage) GetType() ChatMessageType
type TextContent ¶ added in v0.1.3
type TextContent struct {
Text string
}
TextContent is content with some text.
func TextPart ¶ added in v0.1.4
func TextPart(s string) TextContent
TextPart creates TextContent from a given string.
func (TextContent) MarshalJSON ¶ added in v0.1.3
func (tc TextContent) MarshalJSON() ([]byte, error)
func (TextContent) String ¶ added in v0.1.10
func (tc TextContent) String() string
func (*TextContent) UnmarshalJSON ¶ added in v0.1.11
func (tc *TextContent) UnmarshalJSON(data []byte) error
type Tool ¶ added in v0.1.8
type Tool struct { // Type is the type of the tool. Type string `json:"type"` // Function is the function to call. Function *FunctionDefinition `json:"function,omitempty"` }
Tool is a tool that can be used by the model.
type ToolCall ¶ added in v0.1.8
type ToolCall struct { // ID is the unique identifier of the tool call. ID string `json:"id"` // Type is the type of the tool call. Typically, this would be "function". Type string `json:"type"` // FunctionCall is the function call to be executed. FunctionCall *FunctionCall `json:"function,omitempty"` }
ToolCall is a call to a tool (as requested by the model) that should be executed.
func (ToolCall) MarshalJSON ¶ added in v0.1.11
func (*ToolCall) UnmarshalJSON ¶ added in v0.1.11
type ToolCallResponse ¶ added in v0.1.8
type ToolCallResponse struct { // ToolCallID is the ID of the tool call this response is for. ToolCallID string `json:"tool_call_id"` // Name is the name of the tool that was called. Name string `json:"name"` // Content is the textual content of the response. Content string `json:"content"` }
ToolCallResponse is the response returned by a tool call.
func (ToolCallResponse) MarshalJSON ¶ added in v0.1.11
func (tc ToolCallResponse) MarshalJSON() ([]byte, error)
func (*ToolCallResponse) UnmarshalJSON ¶ added in v0.1.11
func (tc *ToolCallResponse) UnmarshalJSON(data []byte) error
type ToolChatMessage ¶ added in v0.1.9
type ToolChatMessage struct { // ID is the ID of the tool call. ID string `json:"tool_call_id"` // Content is the content of the tool message. Content string `json:"content"` }
ToolChatMessage is a chat message representing the result of a tool call.
func (ToolChatMessage) GetContent ¶ added in v0.1.9
func (m ToolChatMessage) GetContent() string
func (ToolChatMessage) GetID ¶ added in v0.1.9
func (m ToolChatMessage) GetID() string
func (ToolChatMessage) GetType ¶ added in v0.1.9
func (m ToolChatMessage) GetType() ChatMessageType
type ToolChoice ¶ added in v0.1.8
type ToolChoice struct { // Type is the type of the tool. Type string `json:"type"` // Function is the function to call (if the tool is a function). Function *FunctionReference `json:"function,omitempty"` }
ToolChoice is a specific tool to use.
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
Package cache provides a generic wrapper that adds caching to a `llms.Model`.
|
Package cache provides a generic wrapper that adds caching to a `llms.Model`. |
Package ernie wrapper around the Baidu Large Language Model Platform APIs.
|
Package ernie wrapper around the Baidu Large Language Model Platform APIs. |
package googleai implements a langchaingo provider for Google AI LLMs.
|
package googleai implements a langchaingo provider for Google AI LLMs. |
internal/cmd
Code generator for vertex.go from googleai.go nolint
|
Code generator for vertex.go from googleai.go nolint |
palm
package palm implements a langchaingo provider for Google Vertex AI legacy PaLM models.
|
package palm implements a langchaingo provider for Google Vertex AI legacy PaLM models. |
vertex
package vertex implements a langchaingo provider for Google Vertex AI LLMs, including the new Gemini models.
|
package vertex implements a langchaingo provider for Google Vertex AI LLMs, including the new Gemini models. |
internal/localclient
Package localclient provides a client for local LLMs.
|
Package localclient provides a client for local LLMs. |