Documentation
¶
Overview ¶
Package suite implements a performance test suite for Noms, intended for measuring and reporting long running tests.
Usage is similar to testify's suite:
- Define a test suite struct which inherits from suite.PerfSuite.
- Define methods on that struct that start with the word "Test", optionally followed by digits, then followed a non-empty capitalized string.
- Call suite.Run with an instance of that struct.
- Run go test with the -perf <path to noms db> flag.
Flags:
-perf.mem Backs the database by a memory store, instead of nbs.
-perf.prefix Gives the dataset IDs for test results a prefix.
-perf.repeat Sets how many times tests are repeated ("reps").
-perf.run Only run tests that match a regex (case insensitive).
-perf.testdata Sets a custom path to the Noms testdata directory.
PerfSuite also supports testify/suite style Setup/TearDown methods:
Setup/TearDownSuite is called exactly once. Setup/TearDownRep is called for each repetition of the test runs, i.e. -perf.repeat times. Setup/TearDownTest is called for every test.
Test results are written to Noms, along with a dump of the environment they were recorded in.
Test names are derived from that "non-empty capitalized string": "Test" is omitted because it's redundant, and leading digits are omitted to allow for manual test ordering. For example:
> cat ./samples/go/csv/csv-import/perf_test.go
type perfSuite {
suite.PerfSuite
}
func (s *perfSuite) TestFoo() { ... }
func (s *perfSuite) TestZoo() { ... }
func (s *perfSuite) Test01Qux() { ... }
func (s *perfSuite) Test02Bar() { ... }
func TestPerf(t *testing.T) {
suite.Run("csv-import", t, &perfSuite{})
}
> noms serve &
> go test -v ./samples/go/csv/... -perf http://localhost:8000 -perf.repeat 3
(perf) RUN(1/3) Test01Qux (recorded as "Qux")
(perf) PASS: Test01Qux (5s, paused 15s, total 20s)
(perf) RUN(1/3) Test02Bar (recorded as "Bar")
(perf) PASS: Test02Bar (15s, paused 2s, total 17s)
(perf) RUN(1/3) TestFoo (recorded as "Foo")
(perf) PASS: TestFoo (10s, paused 1s, total 11s)
(perf) RUN(1/3) TestZoo (recorded as "Zoo")
(perf) PASS: TestZoo (1s, paused 42s, total 43s)
...
> noms show http://localhost:8000::csv-import
{
environment: ...
tests: [{
"Bar": {elapsed: 15s, paused: 2s, total: 17s},
"Foo": {elapsed: 10s, paused: 1s, total: 11s},
"Qux": {elapsed: 5s, paused: 15s, total: 20s},
"Zoo": {elapsed: 1s, paused: 42s, total: 43s},
}, ...]
...
}
Index ¶
- func Run(datasetID string, t *testing.T, suiteT perfSuiteT)
- type PerfSuite
- func (suite *PerfSuite) CloseGlob(files []io.Reader)
- func (suite *PerfSuite) NewAssert() *assert.Assertions
- func (suite *PerfSuite) OpenGlob(pattern ...string) []io.Reader
- func (suite *PerfSuite) Pause(fn func())
- func (suite *PerfSuite) StartRemoteDatabase() (host string, stopFn func())
- func (suite *PerfSuite) Suite() *PerfSuite
- func (suite *PerfSuite) TempDir() string
- func (suite *PerfSuite) TempFile() *os.File
- type SetupRepSuite
- type TearDownRepSuite
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type PerfSuite ¶
type PerfSuite struct {
// T is the testing.T instance set when the suite is passed into Run.
T *testing.T
// W is the io.Writer to write test output, which only outputs if the verbose flag is set.
W io.Writer
// AtticLabs is the path to the attic-labs directory (e.g. /path/to/go/src/github.com/attic-labs).
AtticLabs string
// Testdata is the path to the testdata directory - typically /path/to/go/src/github.com/attic-labs, but it can be overridden with the -perf.testdata flag.
Testdata string
// Database is a Noms database that tests can use for reading and writing. State is persisted across a single Run of a suite.
Database datas.Database
// DatabaseSpec is the Noms spec of Database (typically a localhost URL).
DatabaseSpec string
// contains filtered or unexported fields
}
PerfSuite is the core of the perf testing suite. See package documentation for details.
func (*PerfSuite) NewAssert ¶
func (suite *PerfSuite) NewAssert() *assert.Assertions
NewAssert returns the assert.Assertions instance for this test.
func (*PerfSuite) OpenGlob ¶
OpenGlob opens the concatenation of all files that match pattern, returned as []io.Reader so it can be used immediately with io.MultiReader.
Large CSV files in testdata are broken up into foo.a, foo.b, etc to get around GitHub file size restrictions.
func (*PerfSuite) Pause ¶
func (suite *PerfSuite) Pause(fn func())
Pause pauses the test timer while fn is executing. Useful for omitting long setup code (e.g. copying files) from the test elapsed time.
func (*PerfSuite) StartRemoteDatabase ¶
StartRemoteDatabase creates a new remote database on an arbitrary free port, running on a separate goroutine. Returns the hostname that that database was started on, and a callback to run to shut down the server.
If the -perf.mem flag is specified, the remote database is hosted in memory, not on disk (in a temporary nbs directory).
- Why not use a local database + memory store? Firstly, because the spec would be "mem", and the spec library doesn't know how to reuse stores. Secondly, because it's an unrealistic performance measurement.
- Why use a remote (HTTP) database? It's more realistic to exercise the HTTP stack, even if it's just talking over localhost.
- Why provide an option for nbs vs memory underlying store? Again, nbs is more realistic than memory, and in common cases disk space > memory space. However, on this developer's laptop, there is actually very little disk space, and a lot of memory; plus making the test run a little bit faster locally is nice.
type SetupRepSuite ¶
type SetupRepSuite interface {
SetupRep()
}
SetupRepSuite has a SetupRep method, which runs every repetition of the test, i.e. -perf.repeat times in total.
type TearDownRepSuite ¶
type TearDownRepSuite interface {
TearDownRep()
}
TearDownRepSuite has a TearDownRep method, which runs every repetition of the test, i.e. -perf.repeat times in total.
Source Files