regtest

package
v0.1.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 25, 2022 License: BSD-3-Clause Imports: 33 Imported by: 0

Documentation

Overview

Package regtest provides a framework for writing gopls regression tests.

User reported regressions are often expressed in terms of editor interactions. For example: "When I open my editor in this directory, navigate to this file, and change this line, I get a diagnostic that doesn't make sense". In these cases reproducing, diagnosing, and writing a test to protect against this regression can be difficult.

The regtest package provides an API for developers to express these types of user interactions in ordinary Go tests, validate them, and run them in a variety of execution modes (see gopls/doc/daemon.md for more information on execution modes). This is achieved roughly as follows:

  • the Runner type starts and connects to a gopls instance for each configured execution mode.
  • the Env type provides a collection of resources to use in writing tests (for example a temporary working directory and fake text editor)
  • user interactions with these resources are scripted using test wrappers around the API provided by the golang.org/x/tools/internal/lsp/fake package.

Regressions are expressed in terms of Expectations, which at a high level are conditions that we expect to be met (or not to be met) at some point after performing the interactions in the test. This is necessary because the LSP is by construction asynchronous: both client and server can send eachother notifications without formal acknowledgement that they have been fully processed.

Simple Expectations may be combined to match specific conditions reported by the user. In the example above, a regtest validating that the user-reported bug had been fixed would "expect" that the editor never displays the confusing diagnostic.

Index

Constants

This section is empty.

Variables

View Source
var (
	// InitialWorkspaceLoad is an expectation that the workspace initial load has
	// completed. It is verified via workdone reporting.
	InitialWorkspaceLoad = CompletedWork(lsp.DiagnosticWorkTitle(lsp.FromInitialWorkspaceLoad), 1, false)
)

Functions

func Main

func Main(m *testing.M, hook func(*source.Options))

Main sets up and tears down the shared regtest state.

func Run

func Run(t *testing.T, files string, f TestFunc)

func WithOptions

func WithOptions(opts ...RunOption) configuredRunner

Types

type DiagnosticExpectation

type DiagnosticExpectation struct {
	// contains filtered or unexported fields
}

A DiagnosticExpectation is a condition that must be met by the current set of diagnostics for a file.

func DiagnosticAt

func DiagnosticAt(name string, line, col int) DiagnosticExpectation

DiagnosticAt asserts that there is a diagnostic entry at the position specified by line and col, for the workdir-relative path name.

func NoDiagnosticAt

func NoDiagnosticAt(name string, line, col int) DiagnosticExpectation

NoDiagnosticAt asserts that there is no diagnostic entry at the position specified by line and col, for the workdir-relative path name. This should only be used in combination with OnceMet for a given condition, otherwise it may always succeed.

func NoDiagnosticWithMessage

func NoDiagnosticWithMessage(name, msg string) DiagnosticExpectation

NoDiagnosticWithMessage asserts that there is no diagnostic entry with the given message.

This should only be used in combination with OnceMet for a given condition, otherwise it may always succeed.

func (DiagnosticExpectation) Check

func (e DiagnosticExpectation) Check(s State) Verdict

Check implements the Expectation interface.

func (DiagnosticExpectation) Description

func (e DiagnosticExpectation) Description() string

Description implements the Expectation interface.

type EditorConfig

type EditorConfig fake.EditorConfig

EditorConfig is a RunOption option that configured the regtest editor.

type Env

type Env struct {
	T   testing.TB
	Ctx context.Context

	// Most tests should not need to access the scratch area, editor, server, or
	// connection, but they are available if needed.
	Sandbox *fake.Sandbox
	Editor  *fake.Editor
	Server  servertest.Connector
	// contains filtered or unexported fields
}

Env holds an initialized fake Editor, Workspace, and Server, which may be used for writing tests. It also provides adapter methods that call t.Fatal on any error, so that tests for the happy path may be written without checking errors.

func NewEnv

func NewEnv(ctx context.Context, tb testing.TB, sandbox *fake.Sandbox, ts servertest.Connector, editorConfig fake.EditorConfig, withHooks bool) *Env

NewEnv creates a new test environment using the given scratch environment and gopls server.

func (*Env) AcceptCompletion

func (e *Env) AcceptCompletion(path string, pos fake.Pos, item protocol.CompletionItem)

AcceptCompletion accepts a completion for the given item at the given position.

func (*Env) AnyDiagnosticAtCurrentVersion

func (e *Env) AnyDiagnosticAtCurrentVersion(name string) Expectation

AnyDiagnosticAtCurrentVersion asserts that there is a diagnostic report for the current edited version of the buffer corresponding to the given workdir-relative pathname.

func (*Env) ApplyCodeAction

func (e *Env) ApplyCodeAction(action protocol.CodeAction)

ApplyCodeAction applies the given code action.

func (*Env) ApplyQuickFixes

func (e *Env) ApplyQuickFixes(path string, diagnostics []protocol.Diagnostic)

ApplyQuickFixes processes the quickfix codeAction, calling t.Fatal on any error.

func (*Env) Await

func (e *Env) Await(expectations ...Expectation)

Await waits for all expectations to simultaneously be met. It should only be called from the main test goroutine.

func (*Env) ChangeConfiguration

func (e *Env) ChangeConfiguration(t *testing.T, config *fake.EditorConfig)

func (*Env) ChangeEnv

func (e *Env) ChangeEnv(overlay map[string]string)

ChangeEnv modifies the editor environment and reconfigures the LSP client. TODO: extend this to "ChangeConfiguration", once we refactor the way editor configuration is defined.

func (*Env) ChangeFilesOnDisk

func (e *Env) ChangeFilesOnDisk(events []fake.FileEvent)

func (*Env) CheckForFileChanges

func (e *Env) CheckForFileChanges()

CheckForFileChanges triggers a manual poll of the workspace for any file changes since creation, or since last polling. It is a workaround for the lack of true file watching support in the fake workspace.

func (*Env) CloseBuffer

func (e *Env) CloseBuffer(name string)

CloseBuffer closes an editor buffer without saving, calling t.Fatal on any error.

func (*Env) CodeAction

func (e *Env) CodeAction(path string, diagnostics []protocol.Diagnostic) []protocol.CodeAction

CodeAction calls testDocument/codeAction for the given path, and calls t.Fatal if there are errors.

func (*Env) CodeLens

func (e *Env) CodeLens(path string) []protocol.CodeLens

CodeLens calls textDocument/codeLens for the given path, calling t.Fatal on any error.

func (*Env) Completion

func (e *Env) Completion(path string, pos fake.Pos) *protocol.CompletionList

Completion executes a completion request on the server.

func (*Env) CreateBuffer

func (e *Env) CreateBuffer(name string, content string)

CreateBuffer creates a buffer in the editor, calling t.Fatal on any error.

func (*Env) DiagnosticAtRegexp

func (e *Env) DiagnosticAtRegexp(name, re string) DiagnosticExpectation

DiagnosticAtRegexp expects that there is a diagnostic entry at the start position matching the regexp search string re in the buffer specified by name. Note that this currently ignores the end position.

func (*Env) DiagnosticAtRegexpWithMessage

func (e *Env) DiagnosticAtRegexpWithMessage(name, re, msg string) DiagnosticExpectation

DiagnosticAtRegexpWithMessage is like DiagnosticAtRegexp, but it also checks for the content of the diagnostic message,

func (*Env) DiagnosticsFor

func (e *Env) DiagnosticsFor(name string) *protocol.PublishDiagnosticsParams

DiagnosticsFor returns the current diagnostics for the file. It is useful after waiting on AnyDiagnosticAtCurrentVersion, when the desired diagnostic is not simply described by DiagnosticAt.

func (*Env) DocumentHighlight

func (e *Env) DocumentHighlight(name string, pos fake.Pos) []protocol.DocumentHighlight
func (e *Env) DocumentLink(name string) []protocol.DocumentLink

func (*Env) DoneWithChange

func (e *Env) DoneWithChange() Expectation

DoneWithChange expects all didChange notifications currently sent by the editor to be completely processed.

func (*Env) DoneWithChangeWatchedFiles

func (e *Env) DoneWithChangeWatchedFiles() Expectation

DoneWithChangeWatchedFiles expects all didChangeWatchedFiles notifications currently sent by the editor to be completely processed.

func (*Env) DoneWithClose

func (e *Env) DoneWithClose() Expectation

DoneWithClose expects all didClose notifications currently sent by the editor to be completely processed.

func (*Env) DoneWithOpen

func (e *Env) DoneWithOpen() Expectation

DoneWithOpen expects all didOpen notifications currently sent by the editor to be completely processed.

func (*Env) DoneWithSave

func (e *Env) DoneWithSave() Expectation

DoneWithSave expects all didSave notifications currently sent by the editor to be completely processed.

func (*Env) DumpGoSum

func (e *Env) DumpGoSum(dir string)

DumpGoSum prints the correct go.sum contents for dir in txtar format, for use in creating regtests.

func (*Env) EditBuffer

func (e *Env) EditBuffer(name string, edits ...fake.Edit)

EditBuffer applies edits to an editor buffer, calling t.Fatal on any error.

func (*Env) ExecuteCodeLensCommand

func (e *Env) ExecuteCodeLensCommand(path string, cmd command.Command)

ExecuteCodeLensCommand executes the command for the code lens matching the given command name.

func (*Env) ExecuteCommand

func (e *Env) ExecuteCommand(params *protocol.ExecuteCommandParams, result interface{})

func (*Env) FormatBuffer

func (e *Env) FormatBuffer(name string)

FormatBuffer formats the editor buffer, calling t.Fatal on any error.

func (*Env) GetQuickFixes

func (e *Env) GetQuickFixes(path string, diagnostics []protocol.Diagnostic) []protocol.CodeAction

GetQuickFixes returns the available quick fix code actions.

func (*Env) GoSumDiagnostic

func (e *Env) GoSumDiagnostic(name, module string) Expectation

GoSumDiagnostic asserts that a "go.sum is out of sync" diagnostic for the given module (as formatted in a go.mod file, e.g. "example.com v1.0.0") is present.

func (*Env) GoToDefinition

func (e *Env) GoToDefinition(name string, pos fake.Pos) (string, fake.Pos)

GoToDefinition goes to definition in the editor, calling t.Fatal on any error. It returns the path and position of the resulting jump.

func (*Env) Hover

func (e *Env) Hover(name string, pos fake.Pos) (*protocol.MarkupContent, fake.Pos)

Hover in the editor, calling t.Fatal on any error.

func (*Env) NoDiagnosticAtRegexp

func (e *Env) NoDiagnosticAtRegexp(name, re string) DiagnosticExpectation

NoDiagnosticAtRegexp expects that there is no diagnostic entry at the start position matching the regexp search string re in the buffer specified by name. Note that this currently ignores the end position. This should only be used in combination with OnceMet for a given condition, otherwise it may always succeed.

func (*Env) OpenFile

func (e *Env) OpenFile(name string)

OpenFile opens a file in the editor, calling t.Fatal on any error.

func (*Env) OrganizeImports

func (e *Env) OrganizeImports(name string)

OrganizeImports processes the source.organizeImports codeAction, calling t.Fatal on any error.

func (*Env) ReadWorkspaceFile

func (e *Env) ReadWorkspaceFile(name string) string

ReadWorkspaceFile reads a file from the workspace, calling t.Fatal on any error.

func (*Env) References

func (e *Env) References(path string, pos fake.Pos) []protocol.Location

References calls textDocument/references for the given path at the given position.

func (*Env) RegexpRange

func (e *Env) RegexpRange(name, re string) (fake.Pos, fake.Pos)

RegexpRange returns the range of the first match for re in the buffer specified by name, calling t.Fatal on any error. It first searches for the position in open buffers, then in workspace files.

func (*Env) RegexpReplace

func (e *Env) RegexpReplace(name, regexpStr, replace string)

RegexpReplace replaces the first group in the first match of regexpStr with the replace text, calling t.Fatal on any error.

func (*Env) RegexpSearch

func (e *Env) RegexpSearch(name, re string) fake.Pos

RegexpSearch returns the starting position of the first match for re in the buffer specified by name, calling t.Fatal on any error. It first searches for the position in open buffers, then in workspace files.

func (*Env) RemoveWorkspaceFile

func (e *Env) RemoveWorkspaceFile(name string)

RemoveWorkspaceFile deletes a file on disk but does nothing in the editor. It calls t.Fatal on any error.

func (*Env) Rename added in v0.1.6

func (e *Env) Rename(path string, pos fake.Pos, newName string)

func (*Env) RunGenerate

func (e *Env) RunGenerate(dir string)

RunGenerate runs go:generate on the given dir, calling t.Fatal on any error. It waits for the generate command to complete and checks for file changes before returning.

func (*Env) RunGoCommand

func (e *Env) RunGoCommand(verb string, args ...string)

RunGoCommand runs the given command in the sandbox's default working directory.

func (*Env) RunGoCommandInDir

func (e *Env) RunGoCommandInDir(dir, verb string, args ...string)

RunGoCommandInDir is like RunGoCommand, but executes in the given relative directory of the sandbox.

func (*Env) SaveBuffer

func (e *Env) SaveBuffer(name string)

SaveBuffer saves an editor buffer, calling t.Fatal on any error.

func (*Env) SaveBufferWithoutActions

func (e *Env) SaveBufferWithoutActions(name string)

func (*Env) SetBufferContent

func (e *Env) SetBufferContent(name string, content string)

func (*Env) Symbol

func (e *Env) Symbol(query string) []fake.SymbolInformation

Symbol returns symbols matching query

func (*Env) WorkspaceSymbol added in v0.1.8

func (e *Env) WorkspaceSymbol(sym string) []protocol.SymbolInformation

WorkspaceSymbol calls workspace/symbol

func (*Env) WriteWorkspaceFile

func (e *Env) WriteWorkspaceFile(name, content string)

WriteWorkspaceFile writes a file to disk but does nothing in the editor. It calls t.Fatal on any error.

func (*Env) WriteWorkspaceFiles

func (e *Env) WriteWorkspaceFiles(files map[string]string)

WriteWorkspaceFiles deletes a file on disk but does nothing in the editor. It calls t.Fatal on any error.

type Expectation

type Expectation interface {
	// Check determines whether the state of the editor satisfies the
	// expectation, returning the results that met the condition.
	Check(State) Verdict
	// Description is a human-readable description of the expectation.
	Description() string
}

An Expectation asserts that the state of the editor at a point in time matches an expected condition. This is used for signaling in tests when certain conditions in the editor are met.

func EmptyDiagnostics

func EmptyDiagnostics(name string) Expectation

EmptyDiagnostics asserts that empty diagnostics are sent for the workspace-relative path name.

func EmptyOrNoDiagnostics added in v0.1.5

func EmptyOrNoDiagnostics(name string) Expectation

EmptyOrNoDiagnostics asserts that either no diagnostics are sent for the workspace-relative path name, or empty diagnostics are sent. TODO(rFindley): this subtlety shouldn't be necessary. Gopls should always send at least one diagnostic set for open files.

func NoDiagnostics

func NoDiagnostics(name string) Expectation

NoDiagnostics asserts that no diagnostics are sent for the workspace-relative path name. It should be used primarily in conjunction with a OnceMet, as it has to check that all outstanding diagnostics have already been delivered.

func StartedChange

func StartedChange(i uint64) Expectation

StartedChange expects there to have been i work items started for processing didChange notifications.

type LogExpectation

type LogExpectation struct {
	// contains filtered or unexported fields
}

LogExpectation is an expectation on the log messages received by the editor from gopls.

func LogMatching

func LogMatching(typ protocol.MessageType, re string, count int, atLeast bool) LogExpectation

LogMatching asserts that the client has received a log message of type typ matching the regexp re.

func NoErrorLogs

func NoErrorLogs() LogExpectation

NoErrorLogs asserts that the client has not received any log messages of error severity.

func NoLogMatching

func NoLogMatching(typ protocol.MessageType, re string) LogExpectation

NoLogMatching asserts that the client has not received a log message of type typ matching the regexp re. If re is an empty string, any log message is considered a match.

func (LogExpectation) Check

func (e LogExpectation) Check(s State) Verdict

Check implements the Expectation interface.

func (LogExpectation) Description

func (e LogExpectation) Description() string

Description implements the Expectation interface.

type Mode

type Mode int

Mode is a bitmask that defines for which execution modes a test should run.

const (
	// Singleton mode uses a separate in-process gopls instance for each test,
	// and communicates over pipes to mimic the gopls sidecar execution mode,
	// which communicates over stdin/stderr.
	Singleton Mode = 1 << iota
	// Forwarded forwards connections to a shared in-process gopls instance.
	Forwarded
	// SeparateProcess forwards connection to a shared separate gopls process.
	SeparateProcess
	// Experimental enables all of the experimental configurations that are
	// being developed.
	Experimental
)

func DefaultModes

func DefaultModes() Mode

type RegistrationExpectation

type RegistrationExpectation struct {
	// contains filtered or unexported fields
}

RegistrationExpectation is an expectation on the capability registrations received by the editor from gopls.

func RegistrationMatching

func RegistrationMatching(re string) RegistrationExpectation

RegistrationMatching asserts that the client has received a capability registration matching the given regexp.

func (RegistrationExpectation) Check

Check implements the Expectation interface.

func (RegistrationExpectation) Description

func (e RegistrationExpectation) Description() string

Description implements the Expectation interface.

type RunMultiple

type RunMultiple []struct {
	Name   string
	Runner regtestRunner
}

func (RunMultiple) Run

func (r RunMultiple) Run(t *testing.T, files string, f TestFunc)

type RunOption

type RunOption interface {
	// contains filtered or unexported methods
}

A RunOption augments the behavior of the test runner.

func DebugAddress

func DebugAddress(addr string) RunOption

DebugAddress configures a debug server bound to addr. This option is currently only supported when executing in Singleton mode. It is intended to be used for long-running stress tests.

func GOPROXY

func GOPROXY(goproxy string) RunOption

GOPROXY configures the test environment to have an explicit proxy value. This is intended for stress tests -- to ensure their isolation, regtests should instead use WithProxyFiles.

func InExistingDir

func InExistingDir(dir string) RunOption

InExistingDir runs the test in a pre-existing directory. If set, no initial files may be passed to the runner. It is intended for long-running stress tests.

func InGOPATH

func InGOPATH() RunOption

InGOPATH configures the workspace working directory to be GOPATH, rather than a separate working directory for use with modules.

func LimitWorkspaceScope

func LimitWorkspaceScope() RunOption

LimitWorkspaceScope sets the LimitWorkspaceScope configuration.

func Modes

func Modes(modes Mode) RunOption

Modes configures the execution modes that the test should run in.

func NoDefaultTimeout added in v0.1.9

func NoDefaultTimeout() RunOption

NoDefaultTimeout removes the timeout set by the -regtest_timeout flag, for individual tests that are expected to run longer than is reasonable for ordinary regression tests.

func Options

func Options(hook func(*source.Options)) RunOption

Options configures the various server and user options.

func ProxyFiles

func ProxyFiles(txt string) RunOption

ProxyFiles configures a file proxy using the given txtar-encoded string.

func SendPID

func SendPID() RunOption

func SkipHooks

func SkipHooks(skip bool) RunOption

SkipHooks allows for disabling the test runner's client hooks that are used for instrumenting expectations (tracking diagnostics, logs, work done, etc.). It is intended for performance-sensitive stress tests or benchmarks.

func SkipLogs

func SkipLogs() RunOption

SkipLogs skips the buffering of logs during test execution. It is intended for long-running stress tests.

func WorkspaceFolders

func WorkspaceFolders(relFolders ...string) RunOption

WorkspaceFolders configures the workdir-relative workspace folders to send to the LSP server. By default the editor sends a single workspace folder corresponding to the workdir root. To explicitly configure no workspace folders, use WorkspaceFolders with no arguments.

type Runner

type Runner struct {
	DefaultModes             Mode
	Timeout                  time.Duration
	GoplsPath                string
	PrintGoroutinesOnFailure bool
	TempDir                  string
	SkipCleanup              bool
	OptionsHook              func(*source.Options)
	// contains filtered or unexported fields
}

A Runner runs tests in gopls execution environments, as specified by its modes. For modes that share state (for example, a shared cache or common remote), any tests that execute on the same Runner will share the same state.

func (*Runner) AddCloser

func (r *Runner) AddCloser(closer io.Closer)

AddCloser schedules a closer to be closed at the end of the test run. This is useful for Windows in particular, as

func (*Runner) Close

func (r *Runner) Close() error

Close cleans up resource that have been allocated to this workspace.

func (*Runner) Run

func (r *Runner) Run(t *testing.T, files string, test TestFunc, opts ...RunOption)

Run executes the test function in the default configured gopls execution modes. For each a test run, a new workspace is created containing the un-txtared files specified by filedata.

type SimpleExpectation

type SimpleExpectation struct {
	// contains filtered or unexported fields
}

SimpleExpectation holds an arbitrary check func, and implements the Expectation interface.

func CompletedWork

func CompletedWork(title string, count uint64, atLeast bool) SimpleExpectation

CompletedWork expects a work item to have been completed >= atLeast times.

Since the Progress API doesn't include any hidden metadata, we must use the progress notification title to identify the work we expect to be completed.

func NoOutstandingWork

func NoOutstandingWork() SimpleExpectation

NoOutstandingWork asserts that there is no work initiated using the LSP $/progress API that has not completed.

func NoShowMessage

func NoShowMessage() SimpleExpectation

NoShowMessage asserts that the editor has not received a ShowMessage.

func OnceMet

func OnceMet(precondition Expectation, mustMeets ...Expectation) *SimpleExpectation

OnceMet returns an Expectation that, once the precondition is met, asserts that mustMeet is met.

func OutstandingWork

func OutstandingWork(title, msg string) SimpleExpectation

OutstandingWork expects a work item to be outstanding. The given title must be an exact match, whereas the given msg must only be contained in the work item's message.

func ReadDiagnostics

func ReadDiagnostics(fileName string, into *protocol.PublishDiagnosticsParams) *SimpleExpectation

ReadDiagnostics is an 'expectation' that is used to read diagnostics atomically. It is intended to be used with 'OnceMet'.

func ShowMessageRequest

func ShowMessageRequest(title string) SimpleExpectation

ShowMessageRequest asserts that the editor has received a ShowMessageRequest with an action item that has the given title.

func ShownMessage

func ShownMessage(title string) SimpleExpectation

ShownMessage asserts that the editor has received a ShownMessage with the given title.

func StartedWork

func StartedWork(title string, atLeast uint64) SimpleExpectation

StartedWork expect a work item to have been started >= atLeast times.

See CompletedWork.

func (SimpleExpectation) Check

func (e SimpleExpectation) Check(s State) Verdict

Check invokes e.check.

func (SimpleExpectation) Description

func (e SimpleExpectation) Description() string

Description returns e.descriptin.

type State

type State struct {
	// contains filtered or unexported fields
}

State encapsulates the server state TODO: explain more

func (State) String

func (s State) String() string

type TestFunc

type TestFunc func(t *testing.T, env *Env)

type UnregistrationExpectation

type UnregistrationExpectation struct {
	// contains filtered or unexported fields
}

UnregistrationExpectation is an expectation on the capability unregistrations received by the editor from gopls.

func UnregistrationMatching

func UnregistrationMatching(re string) UnregistrationExpectation

UnregistrationMatching asserts that the client has received an unregistration whose ID matches the given regexp.

func (UnregistrationExpectation) Check

Check implements the Expectation interface.

func (UnregistrationExpectation) Description

func (e UnregistrationExpectation) Description() string

Description implements the Expectation interface.

type Verdict

type Verdict int

A Verdict is the result of checking an expectation against the current editor state.

const (
	// Met indicates that an expectation is satisfied by the current state.
	Met Verdict = iota
	// Unmet indicates that an expectation is not currently met, but could be met
	// in the future.
	Unmet
	// Unmeetable indicates that an expectation cannot be satisfied in the
	// future.
	Unmeetable
)

Order matters for the following constants: verdicts are sorted in order of decisiveness.

func (Verdict) String

func (v Verdict) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL