runstate

Documentation

Overview

Package runstate provides functionality for managing the runtime state of message processing, including message aggregation, forking, and joining of message streams, as well as usage tracking.

Index

• func AddMessage[T messages.ModelMessage](a *Aggregator, m messages.Message[T])
• type AggregatedMessages
  • func (a AggregatedMessages) Len() int
• type Aggregator
  • func (a *Aggregator) AddAssistantMessage(m messages.Message[messages.AssistantMessage])
  • func (a *Aggregator) AddToolCall(m messages.Message[messages.ToolCallMessage])
  • func (a *Aggregator) AddToolResponse(m messages.Message[messages.ToolResponse])
  • func (a *Aggregator) AddUserPrompt(m messages.Message[messages.UserMessage])
  • func (a *Aggregator) Fork() *Aggregator
  • func (a *Aggregator) ID() uuid.UUID
  • func (a *Aggregator) Join(b *Aggregator)
  • func (a *Aggregator) Len() int
  • func (a *Aggregator) Messages() AggregatedMessages
  • func (a *Aggregator) MessagesIter() iter.Seq[messages.Message[messages.ModelMessage]]
  • func (a *Aggregator) Usage() Usage
• type CompletionTokensDetails
  • func (c *CompletionTokensDetails) AddUsage(details *CompletionTokensDetails)
• type PromptTokensDetails
  • func (p *PromptTokensDetails) AddUsage(details *PromptTokensDetails)
• type Usage
  • func (u *Usage) AddUsage(usage *Usage)

Functions

func AddMessage

func AddMessage[T messages.ModelMessage](a *Aggregator, m messages.Message[T])

AddMessage adds any message type that implements ModelMessage to the aggregator. This is a generic function that can handle any valid message type in the system. For common message types, prefer using the specific Add methods (AddUserPrompt, AddAssistantMessage, etc.) as they provide better type safety and clarity.

Example:

agg := &Aggregator{...}
msg := messages.New().UserPrompt("hello")
AddMessage(agg, msg)

Types

type AggregatedMessages

type AggregatedMessages []messages.Message[messages.ModelMessage]

AggregatedMessages represents a collection of model messages that can be processed together. It provides a type-safe way to handle multiple messages while maintaining their order.

func (AggregatedMessages) Len

func (a AggregatedMessages) Len() int

Len returns the number of messages in the collection.

type Aggregator

type Aggregator struct {
	// contains filtered or unexported fields
}

Aggregator manages a collection of messages and their associated usage statistics. It supports fork-join operations to allow parallel processing of message streams while maintaining message order and proper usage tracking.

func (*Aggregator) AddAssistantMessage

func (a *Aggregator) AddAssistantMessage(m messages.Message[messages.AssistantMessage])

AddAssistantMessage adds an assistant's response message to the aggregator. This is used for messages that represent responses or outputs from the assistant.

Example:

msg := messages.New().AssistantMessage("The weather is sunny.")
agg.AddAssistantMessage(msg)

func (*Aggregator) AddToolCall

func (a *Aggregator) AddToolCall(m messages.Message[messages.ToolCallMessage])

AddToolCall adds a tool call message to the aggregator. This is used when the assistant needs to invoke an external tool or service.

Example:

toolCall := messages.New().ToolCall("weather-api", []ToolCallData{...})
agg.AddToolCall(toolCall)

func (*Aggregator) AddToolResponse

func (a *Aggregator) AddToolResponse(m messages.Message[messages.ToolResponse])

AddToolResponse adds a tool's response message to the aggregator. This is used to store the results returned from external tool invocations.

Example:

response := messages.New().ToolResponse("call-id", "weather-api", "Temperature: 72°F")
agg.AddToolResponse(response)

func (*Aggregator) AddUserPrompt

func (a *Aggregator) AddUserPrompt(m messages.Message[messages.UserMessage])

AddUserPrompt adds a user message to the aggregator. This is typically used for adding messages that represent user input or queries.

Example:

msg := messages.New().UserPrompt("What's the weather?")
agg.AddUserPrompt(msg)

func (*Aggregator) Fork

func (a *Aggregator) Fork() *Aggregator

Fork creates a new aggregator that starts with a copy of the current messages. The new aggregator gets: - A new unique ID - A copy of all current messages - An initLen set to the current message count This allows for parallel processing of message streams that can be joined later.

func (*Aggregator) ID

func (a *Aggregator) ID() uuid.UUID

ID returns the unique identifier of this aggregator. This ID is generated when the aggregator is created or forked.

func (*Aggregator) Join

func (a *Aggregator) Join(b *Aggregator)

Join combines messages from a forked aggregator back into this one. It:

• Appends only the messages that were added to the forked aggregator after it was forked (determined using b.initLen)
• Combines usage statistics from both aggregators

The join operation maintains message order by: 1. Keeping all original messages 2. Keeping any messages added to this aggregator after the fork 3. Appending only new messages from the forked aggregator (those after b.initLen)

Example:

original := &Aggregator{...}  // Has messages [1,2]
forked := original.Fork()     // forked has [1,2] and initLen=2
original.Add(msg3)            // original now has [1,2,3]
forked.Add(msg4)             // forked now has [1,2,4]
original.Join(forked)         // original ends with [1,2,3,4]

func (*Aggregator) Len

func (a *Aggregator) Len() int

Len returns the total number of messages currently held by the aggregator.

func (*Aggregator) Messages

func (a *Aggregator) Messages() AggregatedMessages

Messages returns a copy of all messages in the aggregator. The returned slice is a deep copy, so modifications to it won't affect the original messages in the aggregator.

func (*Aggregator) MessagesIter

func (a *Aggregator) MessagesIter() iter.Seq[messages.Message[messages.ModelMessage]]

MessagesIter returns an iterator over all messages in the aggregator. This provides a memory-efficient way to process messages sequentially without creating a copy of the entire message slice.

func (*Aggregator) Usage

func (a *Aggregator) Usage() Usage

Usage returns the current usage statistics for this aggregator. This includes token counts for prompts and completions, as well as detailed breakdowns of token usage by category.

type CompletionTokensDetails

type CompletionTokensDetails struct {
	// When using Predicted Outputs, the number of tokens in the prediction that
	// appeared in the completion.
	AcceptedPredictionTokens int64 `json:"accepted_prediction_tokens"`
	// Audio input tokens generated by the model.
	AudioTokens int64 `json:"audio_tokens"`
	// Tokens generated by the model for reasoning.
	ReasoningTokens int64 `json:"reasoning_tokens"`
	// When using Predicted Outputs, the number of tokens in the prediction that did
	// not appear in the completion. However, like reasoning tokens, these tokens are
	// still counted in the total completion tokens for purposes of billing, output,
	// and context window limits.
	RejectedPredictionTokens int64 `json:"rejected_prediction_tokens"`
}

func (*CompletionTokensDetails) AddUsage

func (c *CompletionTokensDetails) AddUsage(details *CompletionTokensDetails)

type PromptTokensDetails

type PromptTokensDetails struct {
	// Audio input tokens present in the prompt.
	AudioTokens int64 `json:"audio_tokens"`
	// Cached tokens present in the prompt.
	CachedTokens int64 `json:"cached_tokens"`
}

func (*PromptTokensDetails) AddUsage

func (p *PromptTokensDetails) AddUsage(details *PromptTokensDetails)

type Usage

type Usage struct {
	// Number of tokens in the generated completion.
	CompletionTokens int64 `json:"completion_tokens"`
	// Number of tokens in the prompt.
	PromptTokens int64 `json:"prompt_tokens"`
	// Total number of tokens used in the request (prompt + completion).
	TotalTokens int64 `json:"total_tokens"`
	// Breakdown of tokens used in a completion.
	CompletionTokensDetails CompletionTokensDetails `json:"completion_tokens_details"`
	// Breakdown of tokens used in the prompt.
	PromptTokensDetails PromptTokensDetails `json:"prompt_tokens_details"`
}

func (*Usage) AddUsage

func (u *Usage) AddUsage(usage *Usage)