e2e

README

This directory contains end to end tests of LUCI CV.

These tests are designed to run locally with fakes instead of Gerrit, Datastore, Cloud Task Queue.

How to run?

The usual Go way, e.g. go test ./... or go test go.chromium.org/luci/cv/tests/e2e/...

How to debug?

Select failing test, e.g. TestXYZ. Run just this test in verbose mode, which emits copious amounts of logging, as if you are looking at production CV app, except log lines not attributed nor grouped to individual task queue executions.

go test -v -run TestXYZ

To make sense of logs, it may be easiest to save them to a file:

go test -v -run TestXYZ 2>&1 >out.log

If test seems to hang, save your time:

go test -v -run TestXYZ -test.timeout=1s 2>&1 >out.log

If test is flaky, try:

go test -v -run TestXYZ -test.count=1000 -test.failfast 2>&1 >out.log

Guidance on writing new tests

• Actual tests should be added into ..._test.go files, as usual.
• Prefer re-using existing project config setups, instead of generating new ones.
• Mark each TestXYZ func with t.Parallel().
  • Corollary: in rare cases where your test really must not be run in parallel, create a dedicated sub-package for just your test and change global vars as necessary. e2e package is intentionally exportable.
• Helper functions should be added to e2e.go. They should be public (see above).
• Use exactly 1 Convey block inside each TestXYZ function.
• Avoid deeply nested Convey blocks. Ideally there is 1 top level only.
  • Rationale: easier to debug, since each section may produce substantial amount of logs, which are harder to attribute to individual sub-blocks.
  • Corollary: moderate copy-pasta is fine.
• Tests should be fast, finishing with 1s even on under-powered laptops.

Advanced: run with flaky Datastore:

Use special -cv.dsflakiness flag, e.g. go test ./... -cv.dsflakiness=0.06. See flakifyDS function for explanation of what 0.06 value actually means and the shortcomings of the simulation.

Documentation

Overview

Package e2e contains all CV end-to-end tests.

Index

• func MakeCfgCombinable(cgName, gHost, gRepo, gRef string, builders ...*cfgpb.Verifiers_Tryjob_Builder) *cfgpb.Config
• func MakeCfgSingular(cgName, gHost, gRepo, gRef string, builders ...*cfgpb.Verifiers_Tryjob_Builder) *cfgpb.Config
• type Test
  • func (t *Test) AddCommitter(email string)
  • func (t *Test) AddDryRunner(email string)
  • func (t *Test) AddNewPatchsetRunner(email string)
  • func (t *Test) EarliestCreatedRunOf(ctx context.Context, lProject string) *run.Run
  • func (t *Test) ExportedBQAttemptsCount() int
  • func (t *Test) LastMessage(gHost string, gChange int64) *gerritpb.ChangeMessageInfo
  • func (t *Test) LatestRunWithGerritCL(ctx context.Context, gHost string, gChange int64) *run.Run
  • func (t *Test) LoadCL(ctx context.Context, id common.CLID) *changelist.CL
  • func (t *Test) LoadGerritCL(ctx context.Context, gHost string, gChange int64) *changelist.CL
  • func (t *Test) LoadGerritRuns(ctx context.Context, gHost string, gChange int64) []*run.Run
  • func (t *Test) LoadProject(ctx context.Context, lProject string) *prjmanager.Project
  • func (t *Test) LoadRun(ctx context.Context, id common.RunID) *run.Run
  • func (t *Test) LoadRunsOf(ctx context.Context, lProject string) []*run.Run
  • func (t *Test) LogPhase(ctx context.Context, format string, args ...any)
  • func (t *Test) MaxCQVote(ctx context.Context, gHost string, gChange int64) int32
  • func (t *Test) MaxVote(ctx context.Context, gHost string, gChange int64, gLabel string) int32
  • func (t *Test) Now() time.Time
  • func (t *Test) RunEndedPubSubTasks() tqtesting.TaskList
  • func (t *Test) RunUntil(ctx context.Context, stopIf func() bool)
  • func (t *Test) RunUntilT(ctx context.Context, targetTasksCount int, stopIf func() bool)
  • func (t *Test) SetUp(testingT testing.TB) context.Context

Functions

func MakeCfgCombinable

func MakeCfgCombinable(cgName, gHost, gRepo, gRef string, builders ...*cfgpb.Verifiers_Tryjob_Builder) *cfgpb.Config

MakeCfgCombinable return project config with a combinable ConfigGroup.

func MakeCfgSingular

func MakeCfgSingular(cgName, gHost, gRepo, gRef string, builders ...*cfgpb.Verifiers_Tryjob_Builder) *cfgpb.Config

MakeCfgSingular return project config with a single ConfigGroup.

Types

type Test

type Test struct {
	*cvtesting.Test // auto-initialized if nil

	PMNotifier  *prjmanager.Notifier
	RunNotifier *run.Notifier

	AdminServer adminpb.AdminServer
	// contains filtered or unexported fields
}

Test encapsulates e2e setup for a CV test.

Embeds cvtesting.Test, which sets CV's dependencies and some simple CV components (e.g. TreeClient), while this Test focuses on setup of CV's own components.

Typical use:

ct := Test{CVDev: true}
ctx, cancel := ct.SetUp(t)
defer cancel()
...
ct.RunUntil(ctx, func() bool { return len(ct.LoadRunsOf("project")) > 0 })

func (*Test) AddCommitter

func (t *Test) AddCommitter(email string)

AddCommitter adds a given member into the committer group.

func (*Test) AddDryRunner

func (t *Test) AddDryRunner(email string)

AddDryRunner adds a given member into the dry-runner group.

func (*Test) AddNewPatchsetRunner

func (t *Test) AddNewPatchsetRunner(email string)

AddNewPatchsetRunner adds a given member into the new-patchset-runner group.

func (*Test) EarliestCreatedRunOf

func (t *Test) EarliestCreatedRunOf(ctx context.Context, lProject string) *run.Run

EarliestCreatedRunOf returns the earliest created Run in a project.

If there are several such runs, may return any one of them.

Returns nil if there are no Runs.

func (*Test) ExportedBQAttemptsCount

func (t *Test) ExportedBQAttemptsCount() int

ExportedBQAttemptsCount returns number of exported CQ Attempts.

func (*Test) LastMessage

func (t *Test) LastMessage(gHost string, gChange int64) *gerritpb.ChangeMessageInfo

LastMessage returns the last message posted on a Gerrit CL from Gerrit fake.

Returns nil if there are no messages. Panics if the CL doesn't exist.

func (*Test) LatestRunWithGerritCL

func (t *Test) LatestRunWithGerritCL(ctx context.Context, gHost string, gChange int64) *run.Run

LatestRunWithGerritCL returns the latest created Run containing given CL.

If there are several, returns the one with latest .StartTime. Returns nil if there is such Runs, including if Gerrit CL isn't yet in DS.

func (*Test) LoadCL

func (t *Test) LoadCL(ctx context.Context, id common.CLID) *changelist.CL

LoadCL returns CL entity or nil if not exists.

func (*Test) LoadGerritCL

func (t *Test) LoadGerritCL(ctx context.Context, gHost string, gChange int64) *changelist.CL

LoadGerritCL returns CL entity or nil if not exists.

func (*Test) LoadGerritRuns

func (t *Test) LoadGerritRuns(ctx context.Context, gHost string, gChange int64) []*run.Run

LoadGerritRuns loads all Runs from Datastore which include a Gerrit CL.

func (*Test) LoadProject

func (t *Test) LoadProject(ctx context.Context, lProject string) *prjmanager.Project

LoadProject returns Project entity or nil if not exists.

func (*Test) LoadRun

func (t *Test) LoadRun(ctx context.Context, id common.RunID) *run.Run

LoadRun returns Run entity or nil if not exists.

func (*Test) LoadRunsOf

func (t *Test) LoadRunsOf(ctx context.Context, lProject string) []*run.Run

LoadRunsOf loads all Runs of a project from Datastore.

func (*Test) LogPhase

func (t *Test) LogPhase(ctx context.Context, format string, args ...any)

LogPhase emits easy to recognize log like =========================== PHASE: .... ===========================

func (*Test) MaxCQVote

func (t *Test) MaxCQVote(ctx context.Context, gHost string, gChange int64) int32

MaxCQVote returns max CQ vote of a Gerrit CL loaded from Gerrit fake.

Returns 0 if there are no votes. Panics if CL doesn't exist.

func (*Test) MaxVote

func (t *Test) MaxVote(ctx context.Context, gHost string, gChange int64, gLabel string) int32

MaxVote returns max vote of a Gerrit CL loaded from Gerrit fake.

Returns 0 if there are no votes. Panics if CL doesn't exist.

func (*Test) Now

func (t *Test) Now() time.Time

Now returns test clock time in UTC.

func (*Test) RunEndedPubSubTasks

func (t *Test) RunEndedPubSubTasks() tqtesting.TaskList

RunEndedPubSubTasks returns all the succeeded TQ tasks with RunEnded pubsub events.

func (*Test) RunUntil

func (t *Test) RunUntil(ctx context.Context, stopIf func() bool)

RunUntil runs TQ tasks, while stopIf returns false.

If `dsFlakinessFlag` is set, uses flaky datastore for running TQ tasks. If `tqParallelFlag` is set, runs TQ tasks concurrently.

Not goroutine safe.

func (*Test) RunUntilT

func (t *Test) RunUntilT(ctx context.Context, targetTasksCount int, stopIf func() bool)

RunUntilT is the same as RunUntil but with custom approximate number of tasks if ran in serial mode.

Depending on command line test options, allows different number of tasks executions.

func (*Test) SetUp

func (t *Test) SetUp(testingT testing.TB) context.Context

SetUp sets up the end to end test.

Must be called exactly once.