llm

Documentation

Overview

Package llm defines interfaces implemented by LLMs (or LLM-related services).

Index

• Constants
• func EchoJSONResponse(promptParts ...Part) string
• func EchoTextResponse(promptParts ...Part) string
• func UnquoteVector(v Vector) string
• type Blob
• type ContentGenerator
  • func EchoContentGenerator() ContentGenerator
  • func TestContentGenerator(name string, generateContent generateContentFunc) ContentGenerator
• type EmbedDoc
• type Embedder
  • func QuoteEmbedder() Embedder
• type Part
• type PolicyChecker
• type PolicyConfig
  • func AllPolicyTypes() []*PolicyConfig
• type PolicyResult
  • func (pr *PolicyResult) IsViolative() bool
  • func (pr *PolicyResult) String() string
• type PolicyType
• type Schema
• type Text
• type Type
• type Vector
  • func (v *Vector) Decode(enc []byte)
  • func (v Vector) Dot(w Vector) float64
  • func (v Vector) Encode() []byte
• type ViolationResult

Constants

const (
	// Default.
	PolicyTypeUnspecified = PolicyType("POLICY_TYPE_UNSPECIFIED")
	// The model facilitates, promotes or enables access to
	// harmful goods, services, and activities.
	PolicyTypeDangerousContent = PolicyType("DANGEROUS_CONTENT")
	// The model reveals an individual’s personal
	// information and data.
	PolicyTypePIISolicitingReciting = PolicyType("PII_SOLICITING_RECITING")
	// The model generates content that is malicious,
	// intimidating, bullying, or abusive towards another individual.
	PolicyTypeHarassment = PolicyType("HARASSMENT")
	// The model generates content that is sexually
	// explicit in nature.
	PolicyTypeSexuallyExplicit = PolicyType("SEXUALLY_EXPLICIT")
	// The model promotes violence, hatred, discrimination on the
	// basis of race, religion, etc.
	PolicyTypeHateSpeech = PolicyType("HATE_SPEECH")
	// The model provides or offers to facilitate access to
	// medical advice or guidance.
	PolicyTypeMedicalInfo = PolicyType("MEDICAL_INFO")
	// The model generates content that contains
	// gratuitous, realistic descriptions of violence or gore.
	PolicyTypeViolenceAndGore = PolicyType("VIOLENCE_AND_GORE")
	// The model generates profanity and obscenities.
	PolicyTypeObscenityAndProfanity = PolicyType("OBSCENITY_AND_PROFANITY")
)

Possible values for PolicyType.

const (
	// Unspecified result.
	ViolationResultUnspecified = ViolationResult("VIOLATION_RESULT_UNSPECIFIED")
	// The final score is greater or equal the input score
	// threshold.
	ViolationResultViolative = ViolationResult("VIOLATIVE")
	// The final score is smaller than the input score
	// threshold.
	ViolationResultNonViolative = ViolationResult("NON_VIOLATIVE")
	// There was an error and the violation result could
	// not be determined.
	ViolationResultClassificationError = ViolationResult("CLASSIFICATION_ERROR")
)

Possible values for ViolationResult.

Functions

func EchoJSONResponse

func EchoJSONResponse(promptParts ...Part) string

EchoJSONResponse returns the concatenation of the prompt parts, wrapped as a JSON object with a single value "prompt". For testing.

func EchoTextResponse

func EchoTextResponse(promptParts ...Part) string

EchoTextResponse returns the concatenation of the prompt parts. For testing.

func UnquoteVector

func UnquoteVector(v Vector) string

UnquoteVector recovers the original text prefix passed to a QuoteEmbedder's EmbedDocs method. Like QuoteEmbedder, UnquoteVector is only useful in tests.

Types

type Blob

type Blob struct {
	MIMEType string
	Data     []byte
}

A Blob is a Part that is binary data, like an image or video.

type ContentGenerator

type ContentGenerator interface {
	// Model returns the name of the generative model
	// used by this ContentGenerator.
	Model() string
	// GenerateContent generates a text response given a JSON schema
	// and one or more prompt parts.
	// If the JSON schema is nil, GenerateContent outputs a plain text response.
	GenerateContent(ctx context.Context, schema *Schema, parts []Part) (string, error)
	// SetTemperature changes the temperature of the model.
	SetTemperature(float32)
}

A ContentGenerator generates content in response to prompts.

See EchoContentGenerator for a generator, useful for testing, that always responds with a deterministic message derived from the prompts.

See golang.org/x/oscar/internal/gcp/gemini for a real implementation.

func EchoContentGenerator

func EchoContentGenerator() ContentGenerator

EchoContentGenerator returns an implementation of ContentGenerator that responds to Generate calls with responses trivially derived from the prompt.

For testing.

func TestContentGenerator

func TestContentGenerator(name string, generateContent generateContentFunc) ContentGenerator

TestContentGenerator returns a ContentGenerator with the given implementations of [GenerateContent].

This is a convenience function for quickly creating custom test implementations of ContentGenerator.

type EmbedDoc

type EmbedDoc struct {
	Title string // title of document (optional)
	Text  string // text of document
}

An EmbedDoc is a single document to be embedded.

type Embedder

type Embedder interface {
	EmbedDocs(ctx context.Context, docs []EmbedDoc) ([]Vector, error)
}

An Embedder computes vector embeddings of a list of documents.

EmbedDocs accepts an arbitrary number of documents and returns their embeddings. If the underlying implementation has a limit on the batch size, it should make multiple requests in order to process all the documents. If an error occurs after some, but not all, documents have been processed, EmbedDocs can return an error along with a shortened vector slice giving the vectors for a prefix of the document slice.

See QuoteEmbedder for a semantically useless embedder that can nonetheless be helpful when writing tests, and see golang.org/x/oscar/internal/gcp/gemini for a real implementation.

func QuoteEmbedder

func QuoteEmbedder() Embedder

QuoteEmbedder returns an implementation of Embedder that can be useful for testing but is completely pointless for real use. It encodes up to the first 122 bytes of each document directly into the first 122 elements of a 123-element unit vector.

type Part

type Part interface {
	// contains filtered or unexported methods
}

A Part is part of a prompt to a ContentGenerator.

type PolicyChecker

type PolicyChecker interface {
	// Name returns the name of the policy checker.
	// This function should always return the same result for a given
	// instance of a [PolicyChecker].
	Name() string
	// Policies returns the list of policies configured on the checker.
	// This function should always return the same result for a given
	// instance of a [PolicyChecker].
	Policies() []*PolicyConfig
	// CheckText evaluates the policies configured on this [PolicyChecker]
	// against the given text and returns a result for each [PolicyConfig].
	// If the text represents a model output, the prompt parts used to generate it
	// may optionally be provided as context. If the text represents a model input,
	// prompt should be empty.
	CheckText(ctx context.Context, text string, prompt ...Part) ([]*PolicyResult, error)
}

A PolicyChecker checks inputs and outputs to LLMs against a fixed set of safety policies.

type PolicyConfig

type PolicyConfig struct {
	// PolicyType: Required. Type of the policy.
	PolicyType PolicyType
	// Threshold: Optional. Score threshold to use when deciding if the content is
	// violative or non-violative. If not specified, the default 0.5 threshold for
	// the policy will be used.
	Threshold float64
}

A PolicyConfig is a policy to apply to an input or output to an LLM.

Copied from "google.golang.org/api/checks/v1alpha" to avoid direct dependency.

func AllPolicyTypes

func AllPolicyTypes() []*PolicyConfig

AllPolicyTypes returns a list of policies that checks for all available dangerous content types at the default threshold.

type PolicyResult

type PolicyResult struct {
	// PolicyType: Type of the policy.
	PolicyType PolicyType
	// Score: Final score for the results of this policy.
	Score float64
	// ViolationResult: Result of the classification for the policy.
	ViolationResult ViolationResult
}

A PolicyResult is the result of evaluating a policy against an input or output to an LLM.

Copied from "google.golang.org/api/checks/v1alpha" to avoid direct dependency.

func (*PolicyResult) IsViolative

func (pr *PolicyResult) IsViolative() bool

IsViolative reports whether the policy result represents a violated policy.

func (*PolicyResult) String

func (pr *PolicyResult) String() string

String returns a string representation of the policy result.

type PolicyType

type PolicyType string

type Schema

type Schema struct {
	// Required. Data type.
	Type Type
	// Optional. The format of the data. This is used only for primitive
	// datatypes. Supported formats:
	//
	//	for NUMBER type: float, double
	//	for INTEGER type: int32, int64
	Format string
	// Optional. A brief description of the parameter. This could contain examples
	// of use. Parameter description may be formatted as Markdown.
	Description string
	// Optional. Indicates if the value may be null.
	Nullable bool
	// Optional. Possible values of the element of Type.STRING with enum format.
	// For example we can define an Enum Direction as :
	// {type:STRING, format:enum, enum:["EAST", NORTH", "SOUTH", "WEST"]}
	Enum []string
	// Optional. Schema of the elements of Type.ARRAY.
	Items *Schema
	// Optional. Properties of Type.OBJECT.
	Properties map[string]*Schema
	// Optional. Required properties of Type.OBJECT.
	Required []string
}

Schema is the `Schema` object allows the definition of input and output data types. These types can be objects, but also primitives and arrays. Represents a select subset of an [OpenAPI 3.0 schema object](https://spec.openapis.org/oas/v3.0.3#schema).

Copied from github.com/google/generative-ai-go/genai.Schema to avoid a direct dependency on that package.

type Text

type Text string

A Text is a Part that is a string.

type Type

type Type int32

Type contains the list of OpenAPI data types as defined by https://spec.openapis.org/oas/v3.0.3#data-types

Copied from github.com/google/generative-ai-go/genai.Schema to avoid a direct dependency on that package.

const (
	// TypeUnspecified means not specified, should not be used.
	TypeUnspecified Type = 0
	// TypeString means string type.
	TypeString Type = 1
	// TypeNumber means number type.
	TypeNumber Type = 2
	// TypeInteger means integer type.
	TypeInteger Type = 3
	// TypeBoolean means boolean type.
	TypeBoolean Type = 4
	// TypeArray means array type.
	TypeArray Type = 5
	// TypeObject means object type.
	TypeObject Type = 6
)

type Vector

type Vector []float32

A Vector is an embedding vector, typically a high-dimensional unit vector.

func (*Vector) Decode

func (v *Vector) Decode(enc []byte)

Decode decodes the byte encoding enc into the vector v. Enc should be a multiple of 4 bytes; any trailing bytes are ignored.

func (Vector) Dot

func (v Vector) Dot(w Vector) float64

Dot returns the dot product of v and w.

TODO(rsc): Using a float64 for the result is slightly higher precision and may be worth doing in the intermediate calculation but may not be worth the type conversions involved to return a float64. Perhaps the return type should still be float32 even if the math is float64.

func (Vector) Encode

func (v Vector) Encode() []byte

Encode returns a byte encoding of the vector v, suitable for storing in a database.

type ViolationResult

type ViolationResult string