metamorphic

Documentation

Overview

Package metamorphic provides a testing framework for running randomized tests over multiple Pebble databases with varying configurations. Logically equivalent operations should result in equivalent output across all configurations.

Index

• Variables
• func Compare(t TestingT, rootDir string, seed uint64, runDirs []string, ...)
• func CompareHistories(t TestingT, paths []string) (i int, diff string)
• func RunAndCompare(t *testing.T, rootDir string, rOpts ...RunOption)
• func RunOnce(t TestingT, runDir string, seed uint64, historyPath string, ...)
• type CustomOption
• type FailOnMatch
• type InjectErrorsRate
• type KeepData
• type MaxThreads
• type RunOnceOption
• type RunOption
  • func AddCustomRun(name string, serializedOptions string) RunOption
  • func ExtendPreviousRun(opsPath, initialStatePath, initialStateDesc string) RunOption
  • func InnerBinary(path string) RunOption
  • func OpCount(rv randvar.Static) RunOption
  • func ParseCustomTestOption(name string, parseFn func(value string) (CustomOption, bool)) RunOption
  • func RuntimeTrace(name string) RunOption
• type Seed
• type TestOptions
• type TestingT

Variables

var (
	// UseDisk configures RunAndCompare to use the physical filesystem for all
	// generated runs.
	UseDisk = closureOpt(func(ro *runAndCompareOptions) {
		ro.mutateTestOptions = append(ro.mutateTestOptions, func(to *TestOptions) { to.useDisk = true })
	})
	// UseInMemory configures RunAndCompare to use an in-memory virtual
	// filesystem for all generated runs.
	UseInMemory = closureOpt(func(ro *runAndCompareOptions) {
		ro.mutateTestOptions = append(ro.mutateTestOptions, func(to *TestOptions) { to.useDisk = false })
	})
)

Functions

func Compare

func Compare(t TestingT, rootDir string, seed uint64, runDirs []string, rOpts ...RunOnceOption)

Compare runs the metamorphic tests in the provided runDirs and compares their histories.

func CompareHistories

func CompareHistories(t TestingT, paths []string) (i int, diff string)

CompareHistories takes a slice of file paths containing history files. It performs a diff comparing the first path to all other paths. CompareHistories returns the index and diff for the first history that differs. If all the histories are identical, CompareHistories returns a zero index and an empty string.

func RunAndCompare

func RunAndCompare(t *testing.T, rootDir string, rOpts ...RunOption)

RunAndCompare runs the metamorphic tests, using the provided root directory to hold test data.

func RunOnce

func RunOnce(t TestingT, runDir string, seed uint64, historyPath string, rOpts ...RunOnceOption)

RunOnce performs one run of the metamorphic tests. RunOnce expects the directory named by `runDir` to already exist and contain an `OPTIONS` file containing the test run's configuration. The history of the run is persisted to a file at the path `historyPath`.

The `seed` parameter is not functional; it's used for context in logging.

Types

type CustomOption

type CustomOption interface {
	// Name returns the name of the custom option. This is the key under which
	// the option appears in the OPTIONS file, within the [TestOptions] stanza.
	Name() string
	// Value returns the value of the custom option, serialized as it should
	// appear within the OPTIONS file.
	Value() string
	// Close is run after the test database has been closed at the end of the
	// test as well as during restart operations within the test sequence. It's
	// passed a copy of the *pebble.Options. If the custom options hold on to
	// any resources outside, Close should release them.
	Close(*pebble.Options) error
	// Open is run before the test runs and during a restart operation after the
	// test database has been closed and Close has been called. It's passed a
	// copy of the *pebble.Options. If the custom options must acquire any
	// resources before the test continues, it should reacquire them.
	Open(*pebble.Options) error
}

CustomOption defines a custom option that configures the behavior of an individual test run. Like all test options, custom options are serialized to the OPTIONS file even if they're not options ordinarily understood by Pebble.

type FailOnMatch

type FailOnMatch struct {
	*regexp.Regexp
}

FailOnMatch configures the run to fail immediately if the history matches the provided regular expression.

type InjectErrorsRate

type InjectErrorsRate float64

InjectErrorsRate configures the run to inject errors into read-only filesystem operations and retry injected errors.

type KeepData

type KeepData struct{}

KeepData keeps the database directory, even on successful runs. If the test used an in-memory filesystem, the in-memory filesystem will be persisted to the run directory.

type MaxThreads

type MaxThreads int

MaxThreads sets an upper bound on the number of parallel execution threads during replay.

type RunOnceOption

type RunOnceOption interface {
	// contains filtered or unexported methods
}

A RunOnceOption configures the behavior of a single run of the metamorphic tests.

type RunOption

type RunOption interface {
	// contains filtered or unexported methods
}

A RunOption configures the behavior of RunAndCompare.

func AddCustomRun

func AddCustomRun(name string, serializedOptions string) RunOption

AddCustomRun adds an additional run of the metamorphic tests, using the provided OPTIONS file contents. The default options will be used, except those options that are overriden by the provided OPTIONS string.

func ExtendPreviousRun

func ExtendPreviousRun(opsPath, initialStatePath, initialStateDesc string) RunOption

ExtendPreviousRun configures RunAndCompare to use the output of a previous metamorphic test run to seed the this run. It's used in the crossversion metamorphic tests, in which a data directory is upgraded through multiple versions of Pebble, exercising upgrade code paths and cross-version compatibility.

The opsPath should be the filesystem path to the ops file containing the operations run within the previous iteration of the metamorphic test. It's used to inform operation generation to prefer using keys used in the previous run, which are therefore more likely to be "interesting."

The initialStatePath argument should be the filesystem path to the data directory containing the database where the previous run of the metamorphic test left off.

The initialStateDesc argument is presentational and should hold a human-readable description of the initial state.

func InnerBinary

func InnerBinary(path string) RunOption

InnerBinary configures the binary that is called for each run. If not specified, this binary (os.Args[0]) is called.

func OpCount

func OpCount(rv randvar.Static) RunOption

OpCount configures the random variable for the number of operations to generate.

func ParseCustomTestOption

func ParseCustomTestOption(name string, parseFn func(value string) (CustomOption, bool)) RunOption

ParseCustomTestOption adds support for parsing the provided CustomOption from OPTIONS files serialized by the metamorphic tests. This RunOption alone does not cause the metamorphic tests to run with any variant of the provided CustomOption set.

func RuntimeTrace

func RuntimeTrace(name string) RunOption

RuntimeTrace configures each test run to collect a runtime trace and output it with the provided filename.

type Seed

type Seed uint64

Seed configures generation to use the provided seed. Seed may be used to deterministically reproduce the same run.

type TestOptions

type TestOptions struct {
	// Opts holds the *pebble.Options for the test.
	Opts *pebble.Options
	// CustomOptions holds custom test options that are defined outside of this
	// package.
	CustomOpts []CustomOption
	// contains filtered or unexported fields
}

TestOptions describes the options configuring an individual run of the metamorphic tests.

type TestingT

type TestingT interface {
	require.TestingT
	Failed() bool
}

TestingT is an interface wrapper around *testing.T