benchseries

package
v0.0.0-...-9ce2b9b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 9, 2024 License: BSD-3-Clause Imports: 20 Imported by: 0

Documentation

Overview

The benchseries package generates performance differentials and analysis tailored to the output of tools and benchmarks in golang.org/x/benchmarks.

WARNING: The API is experimental, unstable, and subject to change at any time.

Index

Constants

View Source 
const (
	DUPE_REPLACE = iota
	DUPE_COMBINE
)
View Source 
const RFC3339NanoNoZ = "2006-01-02T15:04:05.999999999-07:00"

RFC3339NanoNoZ has the property that formatted date&time.000000000 < date&time.000000001, unlike RFC3339Nano where date&timeZ > date&timeZ.000000001Z i.e., "Z" > "."" but "+" < "." so if ".000000000" is elided must use "+00:00" to express the Z time zone to get the sort right.

Variables

This section is empty.

Functions

func ChangeScore 

func ChangeScore(l1, c1, h1, l2, c2, h2 float64) float64

ChangeScore returns an indicator of the change and direction. This is a heuristic measure of the lack of overlap between two confidence intervals; minimum lack of overlap (i.e., same confidence intervals) is zero. Exact non-overlap, meaning the high end of one interval is equal to the low end of the other, is one. A gap of size G between the two intervals yields a score of 1 + G/M where M is the size of the smaller interval (this penalizes a ChangeScore in noise, which is also a ChangeScore). A partial overlap of size G yields a score of 1 - G/M.

Empty confidence intervals are problematic and produces infinities or NaNs.

func Chart 

func Chart(cs []*ComparisonSeries, pngDir, pdfDir, svgDir string, logScale bool, threshold float64, boring bool)

func NormalizeDateString 

func NormalizeDateString(in string) (string, error)

NormalizeDateString converts dates in two formats used in bent/benchmarking into UTC, so that all sort properly into a single order with no confusion.

func ParseNormalizedDateString 

func ParseNormalizedDateString(in string) (time.Time, error)

ParseNormalizedDateString parses a time in the format returned by NormalizeDateString.

Types

type Builder 

type Builder struct {
	Residues map[benchproc.Key]struct{}
	// contains filtered or unexported fields
}

A Builder collects benchmark results into a set of tables, and transforms that into a slice of ComparisonSeries.

func NewBuilder 

func NewBuilder(bo *BuilderOptions) (*Builder, error)

NewBuilder creates a new Builder for collecting benchmark results into tables. Each result will be mapped to a Table by seriesBy. Within each table, the results are mapped to cells by benchBy and seriesBy. Any results within a single cell that vary by residue will be reported as warnings.

func (*Builder) Add 

func (b *Builder) Add(result *benchfmt.Result)

Add adds all of the values in result to the tables in the Builder.

func (*Builder) AddFiles 

func (b *Builder) AddFiles(files benchfmt.Files) error

func (*Builder) AllComparisonSeries 

func (b *Builder) AllComparisonSeries(existing []*ComparisonSeries, dupeHow int) ([]*ComparisonSeries, error)

AllComparisonSeries converts the accumulated "experiments" into a slice of series of comparisons, with one slice element per goos-goarch-unit. The experiments need not have occurred in any sensible order; this deals with that, including overlaps (depend on flag, either replaces old with younger or combines, REPLACE IS PREFERRED and works properly with combining old summary data with fresh benchmarking data) and possibly also with previously processed summaries.

type BuilderOptions 

type BuilderOptions struct {
	Filter          string // how to filter benchmark results, as a benchproc option (e.g., ".unit:/.*/")
	Series          string // the name of the benchmark key that contains the time of the last commit to the experiment branch (e.g. "numerator_stamp", "tip-commit-time")
	Table           string // list of benchmark keys to group ComparisonSeries tables by, in addition to .unit (e.g., "goarch,goos", "" (none))
	Experiment      string // the name of the benchmark key that contains the time at which the comparative benchmarks were run (e.g., "upload-time", "runstamp")
	Compare         string // the name of the benchmark key that contains the id/role of the toolchain being compared (e.g., "toolchain", "role")
	Numerator       string // the value of the Compare key that indicates the numerator in the ratios (i.e., "test", "tip", "experiment")
	Denominator     string // the value of the Compare key that indicates the denominator in the ratios (i.e., "control", "base", "baseline")
	NumeratorHash   string // the name of the benchmark key that contains the git hash of the numerator (test) toolchain
	DenominatorHash string // the name of the benchmark key that contains the git hash of the denominator (control) toolchain
	Ignore          string // list of benchmark keys to ignore entirely (e.g. "tip,base,bentstamp,suite")
	Warn            func(format string, args ...interface{})
}

func BentBuilderOptions 

func BentBuilderOptions() *BuilderOptions

func DefaultBuilderOptions 

func DefaultBuilderOptions() *BuilderOptions

type Cell 

type Cell struct {
	Values []float64 // Actual values observed for this cell (sorted).  Typically 1-100.

	// Residues is the set of residue Keys mapped to this cell.
	// It is used to check for non-unique keys.
	Residues map[benchproc.Key]struct{}
}

A Cell is the observations for part of a benchmark comparison.

type ChartOptions 

type ChartOptions int

type Comparison 

type Comparison struct {
	Numerator, Denominator *Cell
	Date                   string

	Summary *ComparisonSummary
	// contains filtered or unexported fields
}

A Comparison is a pair of numerator and denominator measurements, the date that they were collected (or the latest date if they were accumulated), an optional slice of medians of ratios of bootstrapped estimates and an optional summary node that contains the spreadsheet/json/database summary of this same information.

type ComparisonHashes 

type ComparisonHashes struct {
	NumHash, DenHash string
}

ComparisonHashes contains the git hashes of the two tool chains being compared.

type ComparisonSeries 

type ComparisonSeries struct {
	Unit string `json:"unit"`

	Benchmarks []string               `json:"benchmarks"`
	Series     []string               `json:"series"`
	Summaries  [][]*ComparisonSummary `json:"summaries"`

	HashPairs map[string]ComparisonHashes `json:"hashpairs"` // maps a series point to the hashes compared at that point.

	Residues []StringAndSlice `json:"residues"`
	// contains filtered or unexported fields
}

A ComparisonSeries describes a table/graph, indexed by paired elements of Benchmarks, Series. Summaries contains the points in the graph. HashPairs includes annotations for the Series axis.

func (*ComparisonSeries) AddSummaries 

func (cs *ComparisonSeries) AddSummaries(confidence float64, N int)

AddSummaries computes the summary data (bootstrapped estimated of the specified confidence interval) for the comparison series cs. The 3rd parameter N specifies the number of sampled bootstraps to use; 1000 is recommended, but 500 is good enough for testing.

func (*ComparisonSeries) ComparisonAt 

func (cs *ComparisonSeries) ComparisonAt(benchmark, series string) (*Comparison, bool)

func (*ComparisonSeries) SummaryAt 

func (cs *ComparisonSeries) SummaryAt(benchmark, series string) (*ComparisonSummary, bool)

func (*ComparisonSeries) ToCsvBootstrapped 

func (cs *ComparisonSeries) ToCsvBootstrapped(out io.Writer, options CsvOptions, threshold float64)

type ComparisonSummary 

type ComparisonSummary struct {
	Low     float64 `json:"low"`
	Center  float64 `json:"center"`
	High    float64 `json:"high"`
	Date    string  `json:"date"`
	Present bool    `json:"present"` // is this initialized?
	// contains filtered or unexported fields
}

A ComparisonSummary is a summary of the comparison of a particular benchmark measurement for two different versions of the toolchain. Low, Center, and High are lower, middle and upper estimates of the value, most likely 2.5%ile, 50%ile, and 97.5%ile from a bootstrap of the original measurement ratios. Date is the (latest) date at which the measurements were taken. Present indicates that Low/Center/High/Date are valid; if comparison is non-nil, then there is a bootstrap that can be used or was used to initialize the other fields. (otherwise the source was JSON or a database).

func (*ComparisonSummary) Defined 

func (s *ComparisonSummary) Defined() bool

func (*ComparisonSummary) HeurOverlap 

func (a *ComparisonSummary) HeurOverlap(b *ComparisonSummary, threshold float64) float64

HeurOverlap computes a heuristic overlap between two confidence intervals

func (*ComparisonSummary) KSov 

func (a *ComparisonSummary) KSov(b *ComparisonSummary) float64

KSov returns the size-adjusted Kolmogorov-Smirnov statistic, equal to D_{n,m} / sqrt((n+m)/n*m). The result can be compared to c(α) where α is the level at which the null hypothesis is rejected. 

   α:  0.2   0.15  0.10  0.05  0.025 0.01  0.005 0.001
c(α):  1.073 1.138 1.224 1.358 1.48  1.628 1.731 1.949

see https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test

type CrossGlyph 

type CrossGlyph struct{}

CrossGlyph is a glyph that draws a big X. this version draws a heavier X.

func (CrossGlyph) DrawGlyph 

func (CrossGlyph) DrawGlyph(c *draw.Canvas, sty draw.GlyphStyle, pt vg.Point)

DrawGlyph implements the Glyph interface.

type CsvOptions 

type CsvOptions int
const (
	CSV_DISTRIBUTION_BITS CsvOptions = 3
	CSV_PLAIN             CsvOptions = 0
	CSV_DELTA             CsvOptions = 1
	CSV_LOHI              CsvOptions = 2

	CSV_VALUES CsvOptions = 4

	CSV_CHANGE_HEU CsvOptions = 8  // This is the interval-overlap heuristic
	CSV_CHANGE_KS  CsvOptions = 16 // This is a Kolmogorov-Smirnov statistic
)

type Lines 

type Lines struct {
	// contains filtered or unexported fields
}

func (Lines) Ticks 

func (u Lines) Ticks(min, max float64) []plot.Tick

type MyBoxPlot 

type MyBoxPlot struct {
	// contains filtered or unexported fields
}

func MyNewBoxPlot 

func MyNewBoxPlot(w vg.Length, loc float64, values plotter.Valuer, moves []directedColor, movers map[int]bool) (*MyBoxPlot, error)

func (*MyBoxPlot) DataRange 

func (b *MyBoxPlot) DataRange() (float64, float64, float64, float64)

func (*MyBoxPlot) GlyphBoxes 

func (b *MyBoxPlot) GlyphBoxes(plt *plot.Plot) []plot.GlyphBox

func (*MyBoxPlot) OutsideLabels 

func (b *MyBoxPlot) OutsideLabels(labels plotter.Labeller) (*plotter.Labels, error)

func (*MyBoxPlot) Plot 

func (p *MyBoxPlot) Plot(c draw.Canvas, plt *plot.Plot)

Plot draws the BoxPlot on Canvas c and Plot plt.

type Point 

type Point struct {
	// contains filtered or unexported fields
}

type SeriesKey 

type SeriesKey struct {
	Benchmark, Series string
}

SeriesKey is a map key used to index a single cell in a ComparisonSeries. ordering is by benchmark, then "series" (== commit) order

type StringAndSlice 

type StringAndSlice struct {
	S     string   `json:"s"`
	Slice []string `json:"slice"`
}

type TriDown 

type TriDown struct{}

func (TriDown) DrawGlyph 

func (TriDown) DrawGlyph(c *draw.Canvas, sty draw.GlyphStyle, pt vg.Point)

DrawGlyph implements the Glyph interface.

type TriUp 

type TriUp struct{}

func (TriUp) DrawGlyph 

func (TriUp) DrawGlyph(c *draw.Canvas, sty draw.GlyphStyle, pt vg.Point)

DrawGlyph implements the Glyph interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.