eval

Documentation

Overview

Package eval lets you evaluate LLM output by scoring it with various scoring methods, and logging the result. It provides a convenient way to run evaluations as part of the standard Go tests using the Run function.

Index

• func Run(t runnerSkipper, name string, f func(e *E))
• type E
  • func (e *E) Log(r Result)
  • func (e *E) Score(s Sample, scorer Scorer) Result
• type Result
• type Sample
• type Score
  • func LevenshteinDistanceScore(s1, s2 string) Score
  • func (s Score) IsValid()
  • func (s Score) String() string
• type Scorer
  • func LevenshteinDistanceScorer() Scorer

Functions

func Run

func Run(t runnerSkipper, name string, f func(e *E))

Run an evaluation. Behaves similar to testing.T.Run, except it skips the test if "go test" is not being run with "-test.run=TestEval". The evaluation function [f] is passed an E to help with scoring, logging, etc.

Types

type E

type E struct {
	T *testing.T
}

func (*E) Log

func (e *E) Log(r Result)

Log a Result.

func (*E) Score

func (e *E) Score(s Sample, scorer Scorer) Result

Score a Sample using a Scorer and return the Result. This is just a convenience method to make it easier to swap out scorers.

type Result

type Result struct {
	Score Score
	Type  string
}

Result of an evaluation with a Score and the type of the Score.

type Sample

type Sample struct {
	Expected string
	Input    string
	Output   string
}

type Score

type Score float64

Score between 0 and 1.

func LevenshteinDistanceScore

func LevenshteinDistanceScore(s1, s2 string) Score

LevenshteinDistanceScore computes a Score between two strings using the levenshtein distance. A score of 1 means the strings are equal, and 0 means they are completely different. Uses https://github.com/agnivade/levenshtein

func (Score) IsValid

func (s Score) IsValid()

func (Score) String

func (s Score) String() string

type Scorer

type Scorer = func(s Sample) Result

Scorer produces a Result with a Score for a Sample.

func LevenshteinDistanceScorer

func LevenshteinDistanceScorer() Scorer

LevenshteinDistanceScorer returns a Scorer that uses the LevenshteinDistanceScore to compare strings.