README
¶
benchparse
benchparse understands Go's benchmark format and parses it into an easy to read structure. The entire spec is defined at https://github.com/golang/proposal/blob/master/design/14313-benchmark-format.md. There are a few subtle parts of the spec that make it less trivial than I thought to parse and conform to correctly.
Usage
Decoding benchmarks
func ExampleDecoder_Decode() {
d := benchparse.Decoder{}
run, err := d.Decode(strings.NewReader(""))
if err != nil {
panic(err)
}
fmt.Println(run)
// Output:
}
Encoding benchmarks
func ExampleEncoder_Encode() {
run := benchparse.Run{
Results:[] benchparse.BenchmarkResult{
{
Name: "BenchmarkBob",
Iterations: 1,
Values: []benchparse.ValueUnitPair{
{
Value: 345,
Unit: "ns/op",
},
},
},
}}
e := benchparse.Encoder{}
if err := e.Encode(os.Stdout, &run); err != nil {
panic(err)
}
// Output: BenchmarkBob 1 345 ns/op
}
Example with changing keys
func ExampleChangingKeys() {
d := benchparse.Decoder{}
run, err := d.Decode(strings.NewReader(`
commit: 7cd9055
BenchmarkDecode/text=digits/level=speed/size=1e4-8 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op
commit: ab322f4
BenchmarkDecode/text=digits/level=speed/size=1e4-8 100 154125 ns/op 64.88 MB/s 40418 B/op 8 allocs/op
`))
if err != nil {
panic(err)
}
fmt.Println("commit of first run", run.Results[0].Configuration.Contents["commit"])
fmt.Println("commit of second run", run.Results[1].Configuration.Contents["commit"])
// Output: commit of first run 7cd9055
// commit of second run ab322f4
}
Example with streaming data
func ExampleDecoder_Stream() {
d := benchparse.Decoder{}
err := d.Stream(context.Background(), strings.NewReader(`
BenchmarkDecode 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op
BenchmarkEncode 100 154125 ns/op 64.88 MB/s 40418 B/op 8 allocs/op
`), func(result benchparse.BenchmarkResult) {
fmt.Println("I got a result named", result.Name)
})
if err != nil {
panic(err)
}
// Output: I got a result named BenchmarkDecode
// I got a result named BenchmarkEncode
}
More complete example
func ExampleRun() {
d := benchparse.Decoder{}
run, err := d.Decode(strings.NewReader(`commit: 7cd9055
BenchmarkDecode/text=digits/level=speed/size=1e4-8 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op
`))
if err != nil {
panic(err)
}
fmt.Println("The number of results:", len(run.Results))
fmt.Println("Git commit:", run.Results[0].Configuration.Contents["commit"])
fmt.Println("Base name of first result:", run.Results[0].BaseName())
fmt.Println("Level config of first result:", run.Results[0].NameAsKeyValue().Contents["level"])
testRunTime, _ := run.Results[0].ValueByUnit(benchparse.UnitRuntime)
fmt.Println("Runtime of first result:", testRunTime)
_, doesMissOpExists := run.Results[0].ValueByUnit("misses/op")
fmt.Println("Does unit misses/op exist in the first run:", doesMissOpExists)
// Output: The number of results: 1
// Git commit: 7cd9055
// Base name of first result: Decode/text=digits/level=speed/size=1e4-8
// Level config of first result: speed
// Runtime of first result: 154125
// Does unit misses/op exist in the first run: false
}
Design Rational
Follows Encode/Encoder/Decode/Decoder pattern of json library. Tries to follow spec strictly since benchmark results can also have extra output. Naming is derived from the proposal's format document. The benchmark will be decoded into a structure like below
// Run is the entire parsed output of a single benchmark run
type Run struct {
// Results are the result of running each benchmark
Results []BenchmarkResult
}
// BenchmarkResult is a single line of a benchmark result
type BenchmarkResult struct {
// Name of this benchmark.
Name string
// Iterations the benchmark run for.
Iterations int
// Values computed by this benchmark. len(Values) >= 1.
Values []ValueUnitPair
// Most benchmarks have the same configuration, but the spec allows a single set of benchmarks to have different
// configurations. Note that as a memory saving feature, multiple BenchmarkResult may share the same Configuration
// data by pointing to the same OrderedStringStringMap. Do not modify the Configuration of any one BenchmarkResult
// unless you are **sure** they do not share the same OrderedStringStringMap data's backing.
Configuration *OrderedStringStringMap
}
// ValueUnitPair is the result of one (of possibly many) benchmark numeric computations
type ValueUnitPair struct {
// Value is the numeric result of a benchmark
Value float64
// Unit is the units this value is in
Unit string
}
// OrderedStringStringMap is a map of strings to strings that maintains ordering.
// This statement implies uniqueness of keys per benchmark.
// "The interpretation of a key/value pair is up to tooling, but the key/value pair is considered to describe all benchmark results that follow, until overwritten by a configuration line with the same key."
type OrderedStringStringMap struct {
// Contents are the values inside this map
Contents map[string]string
// Order is the string order of the contents of this map. It is intended that len(Order) == len(Contents) and the
// keys of Contents are all inside Order.
Order []string
}
A few things about the structure of this spec stick out.
- Benchmarks often have the same configuration but do not have to. It is possible for the benchmark output to change the key/value configuration of a benchmark while running.
- Benchmark keys are stored in an ordered map to make encoding and decoding between benchmark outputs as symetric as possible.
- The implementation had the option to use regex parsing, but since the spec is very clear about exact go functions that should imply deliminators, I use those functions directly.
- There is no strict requirement that the benchmark output contain values for allocations or runtime. There are unit helpers to look these up. For example:
func ExampleBenchmarkResult_ValueByUnit() {
res := &benchparse.BenchmarkResult{
Values: []benchparse.ValueUnitPair {
{
Value: 125,
Unit: "ns/op",
},
},
}
fmt.Println(res.ValueByUnit(benchparse.UnitRuntime))
// Output: 125 true
}
Similar tools
There is a similar tool at https://godoc.org/golang.org/x/tools/benchmark/parse which also parses benchmark output, but does so in a very limited way and not to the flexibility defined by the README spec.
Contributing
Contributions welcome! Submit a pull request on github and make sure your code passes make lint test. For
large changes, I strongly recommend creating an issue on GitHub first to
confirm your change will be accepted before writing a lot of code. GitHub issues are also recommended, at your discretion,
for smaller changes or questions.
License
This library is licensed under the Apache 2.0 License.
Documentation
¶
Overview ¶
Package benchparse allows you to easily parse the output format of Go's benchmark results, as well as other outputs that conform to the benchmark spec. The entire spec is documented at https://github.com/golang/proposal/blob/master/design/14313-benchmark-format.md.
Proper use of this library is to pass a io stream into Decode to decode a benchmark run into results. If you want to modify those results and later encode them back out, make a deep copy of the BenchmarkResult object since by default, Decode will share pointers to Configuration objects.
Index ¶
Examples ¶
Constants ¶
const ( // UnitRuntime is the default unit for Go's runtime benchmark. You're intended to call it with ValueByUnit. UnitRuntime = "ns/op" // UnitBytesAlloc is the default unit for Go's memory allocated benchmark. You're intended to call it with ValueByUnit. UnitBytesAlloc = "B/op" // UnitObjectAllocs is the default unit for Go's # of allocs benchmark. You're intended to call it with ValueByUnit. UnitObjectAllocs = "allocs/op" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BenchmarkResult ¶
type BenchmarkResult struct {
// Name of this benchmark.
Name string
// Iterations the benchmark run for.
Iterations int
// Values computed by this benchmark. len(Values) >= 1.
Values []ValueUnitPair
// Most benchmarks have the same configuration, but the spec allows a single set of benchmarks to have different
// configurations. Note that as a memory saving feature, multiple BenchmarkResult may share the same Configuration
// data by pointing to the same OrderedStringStringMap. Do not modify the Configuration of any one BenchmarkResult
// unless you are **sure** they do not share the same OrderedStringStringMap data's backing.
Configuration *OrderedStringStringMap
}
BenchmarkResult is a single line of a benchmark result
func (BenchmarkResult) AllKeyValuePairs ¶ added in v0.2.0
func (b BenchmarkResult) AllKeyValuePairs() *OrderedStringStringMap
AllKeyValuePairs returns the combination of the configuration key/value pairs followed by the benchmark name's key/value pairs. It handles the special case of -N at the end of the last benchmark key/value pair by removing anything matching "-(\d+)" from the last key/value pair of the benchmark name.
Example ¶
package main
import (
"fmt"
"github.com/cep21/benchparse"
)
func main() {
b := benchparse.BenchmarkResult{
Configuration: &benchparse.OrderedStringStringMap{
Contents: map[string]string{"commit": "a3abd32"},
Order: []string{"commit"},
},
Name: "BenchmarkDecode/text=digits/level=speed/size=1e4-8",
}
fmt.Println(b.AllKeyValuePairs().Contents["size"])
fmt.Println(b.AllKeyValuePairs().Contents["commit"])
}
Output: 1e4 a3abd32
func (BenchmarkResult) NameAsKeyValue ¶
func (b BenchmarkResult) NameAsKeyValue() *OrderedStringStringMap
NameAsKeyValue parses the name of the benchmark as a subtest/subbench split by / assuming you use key=value naming for each sub test. One expected format may be "BenchmarkQuery/runs=1000/dist=normal". For pairs that do not contain a =, like "BenchmarkQuery" above, they will be stored inside OrderedStringStringMap with the key as their name and an empty value. If multiple keys are used (which is not recommended), then the last key's value will be returned.
Note that there is one special case handling. Many go benchmarks append a "-N" number to the end of the benchmark name. This can throw off key handling. If you want to ignore this, you'll have to check the last value in your returned map.
Example ¶
package main
import (
"fmt"
"github.com/cep21/benchparse"
)
func main() {
b := benchparse.BenchmarkResult{
Name: "BenchmarkDecode/text=digits/level=speed/size=1e4-8",
}
fmt.Println(b.NameAsKeyValue().Contents["text"])
}
Output: digits
func (BenchmarkResult) String ¶
func (b BenchmarkResult) String() string
func (BenchmarkResult) ValueByUnit ¶
func (b BenchmarkResult) ValueByUnit(unit string) (float64, bool)
ValueByUnit returns the first value associated with a unit. Returns false if the unit did not exist.
Example ¶
package main
import (
"fmt"
"github.com/cep21/benchparse"
)
func main() {
res := &benchparse.BenchmarkResult{
Values: []benchparse.ValueUnitPair{
{
Value: 125,
Unit: "ns/op",
},
},
}
fmt.Println(res.ValueByUnit(benchparse.UnitRuntime))
}
Output: 125 true
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder helps configure how to decode benchmark results.
func (Decoder) Decode ¶
Decode an input stream into a benchmark run. Returns an error if there are any issues decoding the benchmark, for example from reading from in. The returned run is **NOT** intended to be modified. It contains public members for API convenience, and will share OrderedStringStringMap values to reduce memory allocations. Do not modify the returned Run and expect it to do anything you are wanting it to do. Instead, create your own Run object and assign values to it as you want.
Example ¶
package main
import (
"fmt"
"strings"
"github.com/cep21/benchparse"
)
func main() {
d := benchparse.Decoder{}
run, err := d.Decode(strings.NewReader(""))
if err != nil {
panic(err)
}
fmt.Println(len(run.Results))
}
Output: 0
Example (Changingkeys) ¶
package main
import (
"fmt"
"strings"
"github.com/cep21/benchparse"
)
func main() {
d := benchparse.Decoder{}
run, err := d.Decode(strings.NewReader(`
commit: 7cd9055
BenchmarkDecode/text=digits/level=speed/size=1e4-8 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op
commit: ab322f4
BenchmarkDecode/text=digits/level=speed/size=1e4-8 100 154125 ns/op 64.88 MB/s 40418 B/op 8 allocs/op
`))
if err != nil {
panic(err)
}
fmt.Println("commit of first run", run.Results[0].Configuration.Contents["commit"])
fmt.Println("commit of second run", run.Results[1].Configuration.Contents["commit"])
}
Output: commit of first run 7cd9055 commit of second run ab322f4
Example (Complete) ¶
package main
import (
"fmt"
"strings"
"github.com/cep21/benchparse"
)
func main() {
d := benchparse.Decoder{}
run, err := d.Decode(strings.NewReader(`commit: 7cd9055
BenchmarkDecode/text=digits/level=speed/size=1e4-8 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op
`))
if err != nil {
panic(err)
}
fmt.Println("The number of results:", len(run.Results))
fmt.Println("Git commit:", run.Results[0].Configuration.Contents["commit"])
fmt.Println("Name of first benchmark:", run.Results[0].Name)
fmt.Println("Level config of first result:", run.Results[0].NameAsKeyValue().Contents["level"])
testRunTime, _ := run.Results[0].ValueByUnit(benchparse.UnitRuntime)
fmt.Println("Runtime of first result:", testRunTime)
_, doesMissOpExists := run.Results[0].ValueByUnit("misses/op")
fmt.Println("Does unit misses/op exist in the first run:", doesMissOpExists)
}
Output: The number of results: 1 Git commit: 7cd9055 Name of first benchmark: BenchmarkDecode/text=digits/level=speed/size=1e4-8 Level config of first result: speed Runtime of first result: 154125 Does unit misses/op exist in the first run: false
func (Decoder) Stream ¶
func (d Decoder) Stream(ctx context.Context, in io.Reader, onResult func(result BenchmarkResult)) error
Stream allows live processing of benchmarks. onResult is executed on each BenchmarkResult. Since context isn't part of io.Reader, context is respected between reads from the input stream. See Decode for more complete documentation
Example ¶
package main
import (
"context"
"fmt"
"strings"
"github.com/cep21/benchparse"
)
func main() {
d := benchparse.Decoder{}
err := d.Stream(context.Background(), strings.NewReader(`
BenchmarkDecode 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op
BenchmarkEncode 100 154125 ns/op 64.88 MB/s 40418 B/op 8 allocs/op
`), func(result benchparse.BenchmarkResult) {
fmt.Println("I got a result named", result.Name)
})
if err != nil {
panic(err)
}
}
Output: I got a result named BenchmarkDecode I got a result named BenchmarkEncode
type Encoder ¶
type Encoder struct {
}
Encoder allows converting a Run object back into a format defined by the benchmark spec.
func (*Encoder) Encode ¶
Example ¶
package main
import (
"os"
"github.com/cep21/benchparse"
)
func main() {
run := benchparse.Run{
Results: []benchparse.BenchmarkResult{
{
Name: "BenchmarkBob",
Iterations: 1,
Values: []benchparse.ValueUnitPair{
{
Value: 345,
Unit: "ns/op",
},
},
},
}}
e := benchparse.Encoder{}
if err := e.Encode(os.Stdout, &run); err != nil {
panic(err)
}
}
Output: BenchmarkBob 1 345 ns/op
type OrderedStringStringMap ¶
type OrderedStringStringMap struct {
// Contents are the values inside this map
Contents map[string]string
// Order is the string order of the contents of this map. It is intended that len(Order) == len(Contents) and the
// keys of Contents are all inside Order.
Order []string
}
OrderedStringStringMap is a map of strings to strings that maintains ordering. Ordering allows symmetric encode/decode operations of a benchmark run. Plus, ordering is not strictly mentioned as unimportant in the spec. This statement implies uniqueness of keys per benchmark. "The interpretation of a key/value pair is up to tooling, but the key/value pair is considered to describe all benchmark results that follow, until overwritten by a configuration line with the same key."
Example ¶
package main
import (
"fmt"
"strings"
"github.com/cep21/benchparse"
)
func main() {
d := benchparse.Decoder{}
run, err := d.Decode(strings.NewReader(`
commit: 7cd9055
justthekey:
BenchmarkDecode/text=digits/level=speed/size=1e4-8 100 154125 ns/op 64.88 MB/s 40418 B/op 7 allocs/op
`))
if err != nil {
panic(err)
}
fmt.Println(run.Results[0].Configuration.Contents["commit"])
fmt.Println(run.Results[0].Configuration.Contents["justthekey"])
fmt.Println(run.Results[0].Configuration.Contents["does not exist"])
}
Output: 7cd9055
type Run ¶
type Run struct {
// Results are the result of running each benchmark
Results []BenchmarkResult
}
Run is the entire parsed output of a single benchmark run
type ValueUnitPair ¶
type ValueUnitPair struct {
// Value is the numeric result of a benchmark
Value float64
// Unit is the units this value is in
Unit string
}
ValueUnitPair is the result of one (of possibly many) benchmark numeric computations
func (ValueUnitPair) String ¶
func (b ValueUnitPair) String() string
Source Files