testscript

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 14, 2024 License: BSD-3-Clause Imports: 30 Imported by: 0

Documentation

Overview

Package testscript provides support for defining filesystem-based tests by creating scripts in a directory.

To invoke the tests, call testscript.Run. For example: 

func TestFoo(t *testing.T) {
	testscript.Run(t, testscript.Params{
		Dir: "testdata",
	})
}

A testscript directory holds test scripts with extension txtar or txt run during 'go test'. Each script defines a subtest; the exact set of allowable commands in a script are defined by the parameters passed to the Run function. To run a specific script foo.txtar or foo.txt, run 

go test cmd/go -run=TestName/^foo$

where TestName is the name of the test that Run is called from.

To define an executable command (or several) that can be run as part of the script, call RunMain with the functions that implement the command's functionality. The command functions will be called in a separate process, so are free to mutate global variables without polluting the top level test binary. 

func TestMain(m *testing.M) {
	os.Exit(testscript.RunMain(m, map[string] func() int{
		"testscript": testscriptMain,
	}))
}

In general script files should have short names: a few words, not whole sentences. The first word should be the general category of behavior being tested, often the name of a subcommand to be tested or a concept (vendor, pattern).

Each script is a text archive (go doc golang.org/x/tools/txtar). The script begins with an actual command script to run followed by the content of zero or more supporting files to create in the script's temporary file system before it starts executing.

As an example: 

# hello world
exec cat hello.text
stdout 'hello world\n'
! stderr .

-- hello.text --
hello world

Each script runs in a fresh temporary work directory tree, available to scripts as $WORK. Scripts also have access to these other environment variables: 

PATH=<actual PATH>
HOME=/no-home (USERPROFILE on windows, home on plan9)
TMPDIR=$WORK/.tmp (TMP on windows)
devnull=<value of os.DevNull>
/=<value of os.PathSeparator>
:=<value of os.PathListSeparator>
$=$

The environment variable $exe (lowercase) is an empty string on most systems, ".exe" on Windows.

The script's supporting files are unpacked relative to $WORK and then the script begins execution in that directory as well. Thus the example above runs in $WORK with $WORK/hello.txtar containing the listed contents.

The lines at the top of the script are a sequence of commands to be executed by a small script engine in the testscript package (not the system shell). The script stops and the overall test fails if any particular command fails.

Each line is parsed into a sequence of space-separated command words, with environment variable expansion and # marking an end-of-line comment. Adding single quotes around text keeps spaces in that text from being treated as word separators and also disables environment variable expansion. Inside a single-quoted block of text, a repeated single quote indicates a literal single quote, as in: 

'Don''t communicate by sharing memory.'

A line beginning with # is a comment and conventionally explains what is being done or tested at the start of a new phase in the script.

A special form of environment variable syntax can be used to quote regexp metacharacters inside environment variables. The "@R" suffix is special, and indicates that the variable should be quoted. 

${VAR@R}

The command prefix ! indicates that the command on the rest of the line (typically go or a matching predicate) must fail, not succeed. Only certain commands support this prefix. They are indicated below by [!] in the synopsis.

The command prefix [cond] indicates that the command on the rest of the line should only run when the condition is satisfied. The predefined conditions are:

  • [short] for testing.Short()
  • net for whether the external network can be used
  • [link] for whether the OS has hard link support
  • [symlink] for whether the OS has symbolic link support
  • [exec:prog] for whether prog is available for execution (found by exec.LookPath)
  • [gc] for whether Go was built with gc
  • [gccgo] for whether Go was built with gccgo
  • [go1.x] for whether the Go version is 1.x or later
  • [unix] for whether the OS is Unix-like (that is, would match the 'unix' build constraint)

Any known values of GOOS and GOARCH can also be used as conditions. They will be satisfied if the target OS or architecture match the specified value. For example, the condition [darwin] is true if GOOS=darwin, and [amd64] is true if GOARCH=amd64.

A condition can be negated: [!short] means to run the rest of the line when testing.Short() is false.

Additional conditions can be added by passing a function to Params.Condition.

The predefined commands are:

  • cd dir Change to the given directory for future commands.

  • chmod perm path... Change the permissions of the files or directories named by the path arguments to the given octal mode (000 to 777).

  • [!] cmp file1 file2 Check that the named files have (or do not have) the same content. By convention, file1 is the actual data and file2 the expected data. File1 can be "stdout" or "stderr" to use the standard output or standard error from the most recent exec or wait command. (If the files have differing content and the command is not negated, the failure prints a diff.)

  • [!] cmpenv file1 file2 Like cmp, but environment variables in file2 are substituted before the comparison. For example, $GOOS is replaced by the target GOOS.

  • cp src... dst Copy the listed files to the target file or existing directory. src can include "stdout" or "stderr" to use the standard output or standard error from the most recent exec or go command.

  • env [key=value...] With no arguments, print the environment (useful for debugging). Otherwise add the listed key=value pairs to the environment.

  • [!] exec program [args...] [&] Run the given executable program with the arguments. It must (or must not) succeed. Note that 'exec' does not terminate the script (unlike in Unix shells).

    If the last token is '&', the program executes in the background. The standard output and standard error of the previous command is cleared, but the output of the background process is buffered — and checking of its exit status is delayed — until the next call to 'wait', 'skip', or 'stop' or the end of the test. At the end of the test, any remaining background processes are terminated using os.Interrupt (if supported) or os.Kill.

    If the last token is '&word&` (where "word" is alphanumeric), the command runs in the background but has a name, and can be waited for specifically by passing the word to 'wait', or used to terminate the process by invoking 'kill' with the word passed to it.

    Standard input can be provided using the stdin command; this will be cleared after exec has been called.

  • [!] exists [-readonly] file... Each of the listed files or directories must (or must not) exist. If -readonly is given, the files or directories must be unwritable.

  • [!] grep [-count=N] pattern file The file's content must (or must not) match the regular expression pattern. For positive matches, -count=N specifies an exact number of matches to require.

  • kill [-SIGNAL] [command] Terminate all 'exec' and 'go' commands started in the background (with the '&' token) by sending an termination signal. Recognized signals are KILL and INT. If no signal is specified, KILL is sent.

    If a command argument is specified, it terminates only that command, which must have been started with the final token '&command&` as described for the exec command.

  • mkdir path... Create the listed directories, if they do not already exists.

  • mv path1 path2 Rename path1 to path2. OS-specific restrictions may apply when path1 and path2 are in different directories.

  • rm file... Remove the listed files or directories.

  • skip [message] Mark the test skipped, including the message if given.

  • [!] stderr [-count=N] pattern Apply the grep command (see above) to the standard error from the most recent exec or wait command.

  • stdin file Set the standard input for the next exec command to the contents of the given file. File can be "stdout" or "stderr" to use the standard output or standard error from the most recent exec or wait command.

  • [!] stdout [-count=N] pattern Apply the grep command (see above) to the standard output from the most recent exec or wait command.

  • ttyin [-stdin] file Attach the next exec command to a controlling pseudo-terminal, and use the contents of the given file as the raw terminal input. If -stdin is specified, also attach the terminal to standard input. Note that this does not attach the terminal to standard output/error.

  • [!] ttyout [-count=N] pattern Apply the grep command (see above) to the raw controlling terminal output from the most recent exec command.

  • stop [message] Stop the test early (marking it as passing), including the message if given.

  • symlink file -> target Create file as a symlink to target. The -> (like in ls -l output) is required.

  • unquote file... Rewrite each file by replacing any leading ">" characters from each line. This enables a file to contain substrings that look like txtar file markers. See also https://godoc.org/github.com/01ne/go-internal/txtar#Unquote

  • wait [command] Wait for all 'exec' and 'go' commands started in the background (with the '&' token) to exit, and display success or failure status for them. After a call to wait, the 'stderr' and 'stdout' commands will apply to the concatenation of the corresponding streams of the background commands, in the order in which those commands were started.

    If an argument is specified, it waits for just that command, which must have been started with the final token '&command&` as described for the exec command.

When TestScript runs a script and the script fails, by default TestScript shows the execution of the most recent phase of the script (since the last # comment) and only shows the # comments for earlier phases. For example, here is a multi-phase script with a bug in it (TODO: make this example less go-command specific): 

# GOPATH with p1 in d2, p2 in d2
env GOPATH=$WORK/d1${:}$WORK/d2

# build & install p1
env
go install -i p1
! stale p1
! stale p2

# modify p2 - p1 should appear stale
cp $WORK/p2x.go $WORK/d2/src/p2/p2.go
stale p1 p2

# build & install p1 again
go install -i p11
! stale p1
! stale p2

-- $WORK/d1/src/p1/p1.go --
package p1
import "p2"
func F() { p2.F() }
-- $WORK/d2/src/p2/p2.go --
package p2
func F() {}
-- $WORK/p2x.go --
package p2
func F() {}
func G() {}

The bug is that the final phase installs p11 instead of p1. The test failure looks like: 

$ go test -run=Script
--- FAIL: TestScript (3.75s)
    --- FAIL: TestScript/install_rebuild_gopath (0.16s)
        script_test.go:223:
            # GOPATH with p1 in d2, p2 in d2 (0.000s)
            # build & install p1 (0.087s)
            # modify p2 - p1 should appear stale (0.029s)
            # build & install p1 again (0.022s)
            > go install -i p11
            [stderr]
            can't load package: package p11: cannot find package "p11" in any of:
            	/Users/rsc/go/src/p11 (from $GOROOT)
            	$WORK/d1/src/p11 (from $GOPATH)
            	$WORK/d2/src/p11
            [exit status 1]
            FAIL: unexpected go command failure

        script_test.go:73: failed at testdata/script/install_rebuild_gopath.txt:15 in $WORK/gopath/src

FAIL
exit status 1
FAIL	cmd/go	4.875s
$

Note that the commands in earlier phases have been hidden, so that the relevant commands are more easily found, and the elapsed time for a completed phase is shown next to the phase heading. To see the entire execution, use "go test -v", which also adds an initial environment dump to the beginning of the log.

Note also that in reported output, the actual name of the per-script temporary directory has been consistently replaced with the literal string $WORK.

If Params.TestWork is true, it causes each test to log the name of its $WORK directory and other environment variable settings and also to leave that directory behind when it exits, for manual debugging of failing tests: 

$ go test -run=Script -testwork
--- FAIL: TestScript (3.75s)
    --- FAIL: TestScript/install_rebuild_gopath (0.16s)
        script_test.go:223:
            WORK=/tmp/cmd-go-test-745953508/script-install_rebuild_gopath
            GOARCH=
            GOCACHE=/Users/rsc/Library/Caches/go-build
            GOOS=
            GOPATH=$WORK/gopath
            GOROOT=/Users/rsc/go
            HOME=/no-home
            TMPDIR=$WORK/tmp
            exe=

            # GOPATH with p1 in d2, p2 in d2 (0.000s)
            # build & install p1 (0.085s)
            # modify p2 - p1 should appear stale (0.030s)
            # build & install p1 again (0.019s)
            > go install -i p11
            [stderr]
            can't load package: package p11: cannot find package "p11" in any of:
            	/Users/rsc/go/src/p11 (from $GOROOT)
            	$WORK/d1/src/p11 (from $GOPATH)
            	$WORK/d2/src/p11
            [exit status 1]
            FAIL: unexpected go command failure

        script_test.go:73: failed at testdata/script/install_rebuild_gopath.txt:15 in $WORK/gopath/src

FAIL
exit status 1
FAIL	cmd/go	4.875s
$

$ WORK=/tmp/cmd-go-test-745953508/script-install_rebuild_gopath
$ cd $WORK/d1/src/p1
$ cat p1.go
package p1
import "p2"
func F() { p2.F() }
$

See also: https://pkg.go.dev/github.com/01ne/go-internal/gotooltest

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IgnoreMissedCoverage deprecated 

func IgnoreMissedCoverage()

Deprecated: this option is no longer used.

func Run 

func Run(t *testing.T, p Params)

RunDir runs the tests in the given directory. All files in dir with a ".txt" or ".txtar" extension are considered to be test files.

func RunMain 

func RunMain(m TestingM, commands map[string]func() int) (exitCode int)

RunMain should be called within a TestMain function to allow subcommands to be run in the testscript context.

The commands map holds the set of command names, each with an associated run function which should return the code to pass to os.Exit. It's OK for a command function to exit itself, but this may result in loss of coverage information.

When Run is called, these commands are installed as regular commands in the shell path, so can be invoked with "exec" or via any other command (for example a shell script).

For backwards compatibility, the commands declared in the map can be run without "exec" - that is, "foo" will behave like "exec foo". This can be disabled with Params.RequireExplicitExec to keep consistency across test scripts, and to keep separate process executions explicit.

This function returns an exit code to pass to os.Exit, after calling m.Run.

func RunT 

func RunT(t T, p Params)

RunT is like Run but uses an interface type instead of the concrete *testing.T type to make it possible to use testscript functionality outside of go test.

Types

type Env 

type Env struct {
	// WorkDir holds the path to the root directory of the
	// extracted files.
	WorkDir string
	// Vars holds the initial set environment variables that will be passed to the
	// testscript commands.
	Vars []string
	// Cd holds the initial current working directory.
	Cd string
	// Values holds a map of arbitrary values for use by custom
	// testscript commands. This enables Setup to pass arbitrary
	// values (not just strings) through to custom commands.
	Values map[interface{}]interface{}
	// contains filtered or unexported fields
}

Env holds the environment to use at the start of a test script invocation.

func (*Env) Defer 

func (e *Env) Defer(f func())

Defer arranges for f to be called at the end of the test. If Defer is called multiple times, the defers are executed in reverse order (similar to Go's defer statement)

func (*Env) Getenv 

func (e *Env) Getenv(key string) string

Getenv retrieves the value of the environment variable named by the key. It returns the value, which will be empty if the variable is not present.

func (*Env) Setenv 

func (e *Env) Setenv(key, value string)

Setenv sets the value of the environment variable named by the key. It panics if key is invalid.

func (*Env) T 

func (e *Env) T() T

T returns the t argument passed to the current test by the T.Run method. Note that if the tests were started by calling Run, the returned value will implement testing.TB. Note that, despite that, the underlying value will not be of type *testing.T because *testing.T does not implement T.

If Cleanup is called on the returned value, the function will run after any functions passed to Env.Defer.

type Params 

type Params struct {
	// Dir holds the name of the directory holding the scripts.
	// All files in the directory with a .txtar or .txt suffix will be
	// considered as test scripts. By default the current directory is used.
	// Dir is interpreted relative to the current test directory.
	Dir string

	// Files holds a set of script filenames. If Dir is empty and this
	// is non-nil, these files will be used instead of reading
	// a directory.
	Files []string

	// Setup is called, if not nil, to complete any setup required
	// for a test. The WorkDir and Vars fields will have already
	// been initialized and all the files extracted into WorkDir,
	// and Cd will be the same as WorkDir.
	// The Setup function may modify Vars and Cd as it wishes.
	Setup func(*Env) error

	// Condition is called, if not nil, to determine whether a particular
	// condition is true. It's called only for conditions not in the
	// standard set, and may be nil.
	Condition func(cond string) (bool, error)

	// Cmds holds a map of commands available to the script.
	// It will only be consulted for commands not part of the standard set.
	Cmds map[string]func(ts *TestScript, neg bool, args []string)

	// TestWork specifies that working directories should be
	// left intact for later inspection.
	TestWork bool

	// WorkdirRoot specifies the directory within which scripts' work
	// directories will be created. Setting WorkdirRoot implies TestWork=true.
	// If empty, the work directories will be created inside
	// $GOTMPDIR/go-test-script*, where $GOTMPDIR defaults to os.TempDir().
	WorkdirRoot string

	// Deprecated: this option is no longer used.
	IgnoreMissedCoverage bool

	// UpdateScripts specifies that if a `cmp` command fails and its second
	// argument refers to a file inside the testscript file, the command will
	// succeed and the testscript file will be updated to reflect the actual
	// content (which could be stdout, stderr or a real file).
	//
	// The content will be quoted with txtar.Quote if needed;
	// a manual change will be needed if it is not unquoted in the
	// script.
	UpdateScripts bool

	// RequireExplicitExec requires that commands passed to RunMain must be used
	// in test scripts via `exec cmd` and not simply `cmd`. This can help keep
	// consistency across test scripts as well as keep separate process
	// executions explicit.
	RequireExplicitExec bool

	// RequireUniqueNames requires that names in the txtar archive are unique.
	// By default, later entries silently overwrite earlier ones.
	RequireUniqueNames bool

	// ContinueOnError causes a testscript to try to continue in
	// the face of errors. Once an error has occurred, the script
	// will continue as if in verbose mode.
	ContinueOnError bool

	// Deadline, if not zero, specifies the time at which the test run will have
	// exceeded the timeout. It is equivalent to testing.T's Deadline method,
	// and Run will set it to the method's return value if this field is zero.
	Deadline time.Time
}

Params holds parameters for a call to Run.

type T 

type T interface {
	Skip(...interface{})
	Fatal(...interface{})
	Parallel()
	Log(...interface{})
	FailNow()
	Run(string, func(T))
	// Verbose is usually implemented by the testing package
	// directly rather than on the *testing.T type.
	Verbose() bool
}

T holds all the methods of the *testing.T type that are used by testscript.

type TFailed deprecated 

type TFailed interface {
	Failed() bool
}

Deprecated: this type is unused.

type TestScript 

type TestScript struct {
	// contains filtered or unexported fields
}

A TestScript holds execution state for a single test script.

func (*TestScript) BackgroundCmds 

func (ts *TestScript) BackgroundCmds() []*exec.Cmd

BackgroundCmds returns a slice containing all the commands that have been started in the background since the most recent wait command, or the start of the script if wait has not been called.

func (*TestScript) Chdir 

func (ts *TestScript) Chdir(dir string) error

Chdir changes the current directory of the script. The path may be relative to the current directory.

func (*TestScript) Check 

func (ts *TestScript) Check(err error)

Check calls ts.Fatalf if err != nil.

func (*TestScript) Defer 

func (ts *TestScript) Defer(f func())

Defer arranges for f to be called at the end of the test. If Defer is called multiple times, the defers are executed in reverse order (similar to Go's defer statement)

func (*TestScript) Exec 

func (ts *TestScript) Exec(command string, args ...string) error

Exec runs the given command and saves its stdout and stderr so they can be inspected by subsequent script commands.

func (*TestScript) Fatalf 

func (ts *TestScript) Fatalf(format string, args ...interface{})

fatalf aborts the test with the given failure message.

func (*TestScript) Getenv 

func (ts *TestScript) Getenv(key string) string

Getenv gets the value of the environment variable named by the key.

func (*TestScript) Logf 

func (ts *TestScript) Logf(format string, args ...interface{})

Logf appends the given formatted message to the test log transcript.

func (*TestScript) MkAbs 

func (ts *TestScript) MkAbs(file string) string

MkAbs interprets file relative to the test script's current directory and returns the corresponding absolute path.

func (*TestScript) Name 

func (ts *TestScript) Name() string

Name returns the short name or basename of the test script.

func (*TestScript) ReadFile 

func (ts *TestScript) ReadFile(file string) string

ReadFile returns the contents of the file with the given name, interpreted relative to the test script's current directory. It interprets "stdout" and "stderr" to mean the standard output or standard error from the most recent exec or wait command respectively.

If the file cannot be read, the script fails.

func (*TestScript) Setenv 

func (ts *TestScript) Setenv(key, value string)

Setenv sets the value of the environment variable named by the key.

func (*TestScript) Stderr 

func (ts *TestScript) Stderr() io.Writer

Stderr returns an io.Writer that can be used by a user-supplied builtin command (declared via Params.Cmds) to write to stderr. If this method is called outside of the execution of a user-supplied builtin command, the call panics.

func (*TestScript) Stdout 

func (ts *TestScript) Stdout() io.Writer

Stdout returns an io.Writer that can be used by a user-supplied builtin command (declared via Params.Cmds) to write to stdout. If this method is called outside of the execution of a user-supplied builtin command, the call panics.

func (*TestScript) Value 

func (ts *TestScript) Value(key interface{}) interface{}

Value returns a value from Env.Values, or nil if no value was set by Setup.

type TestingM 

type TestingM interface {
	Run() int
}

TestingM is implemented by *testing.M. It's defined as an interface to allow testscript to co-exist with other testing frameworks that might also wish to call M.Run.

Source Files

View all Source files

Directories

Path Synopsis
internal
pty

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.