benchstat

Documentation

Overview

Package benchstat is deprecated.

This package contains the underlying implementation of an old version of the benchstat command.

Deprecated: The latest version of benchstat can be found at golang.org/x/perf/cmd/benchstat. To work with benchmark data, see golang.org/x/perf/benchproc and golang.org/x/perf/benchmath.

Index

• Variables
• func ByDelta(t *Table, i, j int) bool
• func ByName(t *Table, i, j int) bool
• func FormatCSV(w io.Writer, tables []*Table, norange bool)
• func FormatHTML(buf *bytes.Buffer, tables []*Table)
• func FormatText(w io.Writer, tables []*Table)
• func NoDeltaTest(old, new *Metrics) (pval float64, err error)
• func SafeFormatHTML(tables []*Table) safehtml.HTML
• func Sort(t *Table, order Order)
• func TTest(old, new *Metrics) (pval float64, err error)
• func UTest(old, new *Metrics) (pval float64, err error)
• type Collection
  • func (c *Collection) AddConfig(config string, data []byte)
  • func (c *Collection) AddFile(config string, r io.Reader) error
  • func (c *Collection) AddResults(config string, results []*benchfmt.Result)
  • func (c *Collection) Tables() []*Table
• type DeltaTest
• type Key
• type Metrics
  • func (m *Metrics) Format(scaler Scaler) string
  • func (m *Metrics) FormatDiff() string
  • func (m *Metrics) FormatMean(scaler Scaler) string
• type Order
  • func Reverse(order Order) Order
• type Row
• type Scaler
  • func NewScaler(val float64, unit string) Scaler
• type Table

Variables

var (
	ErrSamplesEqual = errors.New("all equal")
	ErrSampleSize   = errors.New("too few samples")
	ErrZeroVariance = errors.New("zero variance")
)

Errors returned by DeltaTest.

Functions

func ByDelta

func ByDelta(t *Table, i, j int) bool

ByDelta sorts tables by the Delta column, reversing the order when larger is better (for "speed" results).

func ByName

func ByName(t *Table, i, j int) bool

ByName sorts tables by the Benchmark name column

func FormatCSV

func FormatCSV(w io.Writer, tables []*Table, norange bool)

FormatCSV appends a CSV formatting of the tables to w. norange suppresses the range columns.

func FormatHTML

func FormatHTML(buf *bytes.Buffer, tables []*Table)

FormatHTML appends an HTML formatting of the tables to buf.

func FormatText

func FormatText(w io.Writer, tables []*Table)

FormatText appends a fixed-width text formatting of the tables to w.

func NoDeltaTest

func NoDeltaTest(old, new *Metrics) (pval float64, err error)

NoDeltaTest applies no delta test; it returns -1, nil.

func SafeFormatHTML

func SafeFormatHTML(tables []*Table) safehtml.HTML

SafeFormatHTML returns the HTML formatting of the tables.

func Sort

func Sort(t *Table, order Order)

Sort sorts a Table t (in place) by the given order.

func TTest

func TTest(old, new *Metrics) (pval float64, err error)

TTest is a DeltaTest using the two-sample Welch t-test.

func UTest

func UTest(old, new *Metrics) (pval float64, err error)

UTest is a DeltaTest using the Mann-Whitney U test.

Types

type Collection

type Collection struct {
	// Configs, Groups, and Units give the set of configs,
	// groups, and units from the keys in Stats in an order
	// meant to match the order the benchmarks were read in.
	Configs, Groups, Units []string

	// Benchmarks gives the set of benchmarks from the keys in
	// Stats by group in an order meant to match the order
	// benchmarks were read in.
	Benchmarks map[string][]string

	// Metrics holds the accumulated metrics for each key.
	Metrics map[Key]*Metrics

	// DeltaTest is the test to use to decide if a change is significant.
	// If nil, it defaults to UTest.
	DeltaTest DeltaTest

	// Alpha is the p-value cutoff to report a change as significant.
	// If zero, it defaults to 0.05.
	Alpha float64

	// AddGeoMean specifies whether to add a line to the table
	// showing the geometric mean of all the benchmark results.
	AddGeoMean bool

	// SplitBy specifies the labels to split results by.
	// By default, results will only be split by full name.
	SplitBy []string

	// Order specifies the row display order for this table.
	// If Order is nil, the table rows are printed in order of
	// first appearance in the input.
	Order Order
}

A Collection is a collection of benchmark results.

func (*Collection) AddConfig

func (c *Collection) AddConfig(config string, data []byte)

AddConfig adds the benchmark results in the formatted data to the named configuration. If the input is large, AddFile should be preferred, since it avoids the need to read a copy of the entire raw input into memory.

func (*Collection) AddFile

func (c *Collection) AddFile(config string, r io.Reader) error

AddFile adds the benchmark results in the formatted data (read from the reader r) to the named configuration.

func (*Collection) AddResults

func (c *Collection) AddResults(config string, results []*benchfmt.Result)

AddResults adds the benchmark results to the named configuration.

func (*Collection) Tables

func (c *Collection) Tables() []*Table

Tables returns tables comparing the benchmarks in the collection.

type DeltaTest

type DeltaTest func(old, new *Metrics) (float64, error)

A DeltaTest compares the old and new metrics and returns the expected probability that they are drawn from the same distribution.

If a probability cannot be computed, the DeltaTest returns an error explaining why. Common errors include ErrSamplesEqual (all samples are equal), ErrSampleSize (there aren't enough samples), and ErrZeroVariance (the sample has zero variance).

As a special case, the missing test NoDeltaTest returns -1, nil.

type Key

type Key struct {
	Config, Group, Benchmark, Unit string
}

A Key identifies one metric (e.g., "ns/op", "B/op") from one benchmark (function name sans "Benchmark" prefix) and optional group in one configuration (input file name).

type Metrics

type Metrics struct {
	Unit    string    // unit being measured
	Values  []float64 // measured values
	RValues []float64 // Values with outliers removed
	Min     float64   // min of RValues
	Mean    float64   // mean of RValues
	Max     float64   // max of RValues
}

A Metrics holds the measurements of a single metric (for example, ns/op or MB/s) for all runs of a particular benchmark.

func (*Metrics) Format

func (m *Metrics) Format(scaler Scaler) string

Format returns a textual formatting of "Mean ±Diff" using scaler.

func (*Metrics) FormatDiff

func (m *Metrics) FormatDiff() string

FormatDiff computes and formats the percent variation of max and min compared to mean. If b.Mean or b.Max is zero, FormatDiff returns an empty string.

func (*Metrics) FormatMean

func (m *Metrics) FormatMean(scaler Scaler) string

FormatMean formats m.Mean using scaler.

type Order

type Order func(t *Table, i, j int) bool

An Order defines a sort order for a table. It reports whether t.Rows[i] should appear before t.Rows[j].

func Reverse

func Reverse(order Order) Order

Reverse returns the reverse of the given order.

type Row

type Row struct {
	Benchmark string     // benchmark name
	Group     string     // group name
	Scaler    Scaler     // formatter for stats means
	Metrics   []*Metrics // columns of statistics
	PctDelta  float64    // unformatted percent change
	Delta     string     // formatted percent change
	Note      string     // additional information
	Change    int        // +1 better, -1 worse, 0 unchanged
}

A Row is a table row for display in the benchstat output.

type Scaler

type Scaler func(float64) string

A Scaler is a function that scales and formats a measurement. All measurements within a given table row are formatted using the same scaler, so that the units are consistent across the row.

func NewScaler

func NewScaler(val float64, unit string) Scaler

NewScaler returns a Scaler appropriate for formatting the measurement val, which has the given unit.

type Table

type Table struct {
	Metric      string
	OldNewDelta bool // is this an old-new-delta table?
	Configs     []string
	Groups      []string
	Rows        []*Row
}

A Table is a table for display in the benchstat output.