worker

Documentation

Overview

Package worker implements a NATS-based worker for large language models.

Index

• func Run(ctx context.Context, cf map[string]any, options ...Option) error
• type Option
  • func Conn(conn *nats.Conn) Option
  • func Hook(...) Option
• type Options

Functions

func Run

func Run(ctx context.Context, cf map[string]any, options ...Option) error

Run will run a NATS-based worker with the provided NATS connection and configuration.

Types

type Option

type Option func(*worker)

An Option is a function that alters a worker's behavior.

func Conn

func Conn(conn *nats.Conn) Option

Conn sets the NATS connection to use for getting requests and publishing responses. This is an alternative to letting the worker manage its own connection.

func Hook

func Hook(hook func(ctx context.Context, model llm.Interface, req *msg.PredictRequest) error) Option

A Hook is a function that is called before a prediction is made, allowing it to alter the request. This is useful when a worker can do some preprocessing of input data, such as reducing the input length to fit context.

type Options

type Options struct {
	client.Options

	// LLM implementation to use.
	LLMType string `json:"type"`

	// Dir provides the directory where models are stored, defaults to "."
	Dir string `json:"dir,omitempty"`
}

Options describes the configuration options for a NATS-based worker. Note that this configuration will be passed on to the LLM driver, so the driver may have additional configuration options.