llmapp

Documentation

Overview

Package llmapp provides applications for LLM content generation to complete higher-level tasks.

All functionality is provided by Client, created by New.

Cached LLM responses are stored in the Client's database as:

("llmapp.GenerateText", generativeModel, promptHash) -> [response]

Note that currently there is no clear way to clean up old cache values that are no longer relevant, but we might want to add this in the future.

We can, however, easily delete ALL cache values and start over by deleting all database entries starting with "llmapp.GenerateText".

Index

• func RelatedTestGenerator(t *testing.T, numRelated int) llm.ContentGenerator
• type Client
  • func New(lg *slog.Logger, g llm.ContentGenerator, db storage.DB) *Client
  • func NewWithChecker(lg *slog.Logger, g llm.ContentGenerator, checker llm.PolicyChecker, ...) *Client
  • func (c *Client) AnalyzeRelated(ctx context.Context, doc *Doc, related []*Doc) (*RelatedAnalysis, error)
  • func (c *Client) EvaluatePolicy(ctx context.Context, prompts []llm.Part, output string) *PolicyEvaluation
  • func (c *Client) Overview(ctx context.Context, docs ...*Doc) (*Result, error)
  • func (c *Client) PostOverview(ctx context.Context, post *Doc, comments []*Doc) (*Result, error)
  • func (c *Client) UpdatedPostOverview(ctx context.Context, post *Doc, oldComments, newComments []*Doc) (*Result, error)
• type Doc
• type PolicyEvaluation
  • func (pe *PolicyEvaluation) String() string
• type PolicyResult
  • func (pr *PolicyResult) String() string
• type Related
• type RelatedAnalysis
• type RelatedDoc
• type Result
  • func (r *Result) HasPolicyViolation() bool

Functions

func RelatedTestGenerator

func RelatedTestGenerator(t *testing.T, numRelated int) llm.ContentGenerator

RelatedTestGenerator returns an llm.ContentGenerator that can be used in tests of the Client.AnalyzeRelated method. numRelated is the number of related documents that will be provided to the Client.AnalyzeRelated call.

For testing.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is a client for accessing the LLM application functionality.

func New

func New(lg *slog.Logger, g llm.ContentGenerator, db storage.DB) *Client

New returns a new client. g is the underlying LLM content generator to use, and db is the database to use as a cache.

func NewWithChecker

func NewWithChecker(lg *slog.Logger, g llm.ContentGenerator, checker llm.PolicyChecker, db storage.DB) *Client

NewWithChecker is like New, but it configures the Client to use the given checker to check the inputs to and outputs of the LLM against safety policies.

When any of the Overview functions are called, the prompts and outputs of the LLM will be checked for safety violations.

If the checker is nil, NewWithChecker is identical to New.

func (*Client) AnalyzeRelated

func (c *Client) AnalyzeRelated(ctx context.Context, doc *Doc, related []*Doc) (*RelatedAnalysis, error)

AnalyzeRelated returns a structured, LLM-generated analysis of the given document and related documents. AnalyzeRelated returns an error if no initial document is provided, no related docs are provided, or the LLM is unable to generate a response. TODO(tatianabradley): Return an error if the LLM generates JSON with missing or unexpected elements.

func (*Client) EvaluatePolicy

func (c *Client) EvaluatePolicy(ctx context.Context, prompts []llm.Part, output string) *PolicyEvaluation

EvaluatePolicy invokes the policy checker on the given prompts and LLM output and wraps its results as a *PolicyEvaluation.

func (*Client) Overview

func (c *Client) Overview(ctx context.Context, docs ...*Doc) (*Result, error)

Overview returns an LLM-generated overview of the given documents, styled with markdown. Overview returns an error if no documents are provided or the LLM is unable to generate a response.

func (*Client) PostOverview

func (c *Client) PostOverview(ctx context.Context, post *Doc, comments []*Doc) (*Result, error)

PostOverview returns an LLM-generated overview of the given post and comments, styled with markdown. PostOverview returns an error if no post is provided or the LLM is unable to generate a response.

func (*Client) UpdatedPostOverview

func (c *Client) UpdatedPostOverview(ctx context.Context, post *Doc, oldComments, newComments []*Doc) (*Result, error)

UpdatedPostOverview returns an LLM-generated overview of the given post and comments, styled with markdown. It summarizes the oldComments and newComments separately. UpdatedPostOverview returns an error if no post is provided or the LLM is unable to generate a response.

type Doc

type Doc struct {
	// Freeform text describing the type of document.
	// Used to help the LLM distinguish between kinds of
	// documents, or understand their relative importance.
	Type string `json:"type,omitempty"`
	// The URL of the document, if one exists.
	URL string `json:"url,omitempty"`
	// The author of the document, if known.
	Author string `json:"author,omitempty"`
	// The title of the document, if known.
	Title string `json:"title,omitempty"`
	Text  string `json:"text"` // required
}

A Doc is a document to provide to an LLM as part of a prompt.

type PolicyEvaluation

type PolicyEvaluation struct {
	Violative     bool // whether any violations were found
	PromptResults []*PolicyResult
	OutputResults *PolicyResult
}

A PolicyEvaluation is the result of evaluating a policy against a multi-part prompt and an output of an LLM.

func (*PolicyEvaluation) String

func (pe *PolicyEvaluation) String() string

String returns a human readable representation of a policy evaluation.

type PolicyResult

type PolicyResult struct {
	Text       string // the text that was analyzed
	Results    []*llm.PolicyResult
	Violations []*llm.PolicyResult
	Error      error
	Cached     bool // whether the result was cached
}

A PolicyResult is the result of evaluating a policy against an input or output to an LLM.

func (*PolicyResult) String

func (pr *PolicyResult) String() string

String returns a human readable representation of a policy result.

type Related

type Related struct {
	Summary string       `json:"original_summary"`
	Related []RelatedDoc `json:"related"`
}

Related represents the desired JSON structure of the LLM output requested by [AnalyzeRelated]. See [relatedSchema] for a description of the fields.

IMPORTANT: If you add, remove or edit the types or JSON names of fields in this struct, edit [relatedSchema] and [relatedTestOutput] accordingly.

type RelatedAnalysis

type RelatedAnalysis struct {
	Result
	// The LLM's response, unmarshaled into a Go struct.
	Output Related
}

RelatedAnalysis is the output of [AnalyzeRelated].

type RelatedDoc

type RelatedDoc struct {
	Title           string `json:"title"`
	URL             string `json:"url"`
	Summary         string `json:"summary"`
	Relationship    string `json:"relationship"`
	Relevance       string `json:"relevance"`
	RelevanceReason string `json:"relevance_reason"`
}

RelatedDoc represents the desired JSON structure of the LLM output for a single related document.

type Result

type Result struct {
	Response         string            // the raw LLM-generated response
	Cached           bool              // whether the response was cached
	Schema           *llm.Schema       // the JSON schema used to generate the result (nil if none)
	Prompt           []llm.Part        // the prompt(s) used to generate the result
	PolicyEvaluation *PolicyEvaluation // (if a policy checker is configured) the policy evaluation result
}

Result is the result of an LLM call.

func (*Result) HasPolicyViolation

func (r *Result) HasPolicyViolation() bool

HasPolicyViolation reports whether the result or its prompts have any policy violations.