golangbuild

Documentation

Overview

Binary golangbuild is a luciexe binary that builds and tests the code for the Go project. It supports building and testing go.googlesource.com/go as well as Go project subrepositories (e.g. go.googlesource.com/net) and on different branches.

To build and run this locally end-to-end, follow these steps:

luci-auth login -scopes "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/gerritcodereview https://www.googleapis.com/auth/cloud-platform"
cat > build.jsonpb <<EOF
{
	"builder": {
		"project": "go",
		"bucket": "ci",
		"builder": "linux-amd64"
	},
	"input": {
		"properties": {
			"project": "go",
			"bootstrap_version": "1.20.6",
			"host": {
				"goarch": "amd64",
				"goos": "linux"
			},
			"target": {
				"goarch": "amd64",
				"goos": "linux"
			},
			"git_cache": "gitcache",
			"tools_cache": "tools"
		},
		"gitiles_commit": {
			"host": "go.googlesource.com",
			"project": "go",
			"id": "27301e8247580e456e712a07d68890dc1e857000",
			"ref": "refs/heads/master"
		}
	}
}
EOF
export SWARMING_SERVER=https://chromium-swarm.appspot.com
LUCIEXE_FAKEBUILD=./build.jsonpb golangbuild

Modify `build.jsonpb` as needed in order to try different paths. The format of `build.jsonpb` is a JSON-encoded protobuf with schema `go.chromium.org/luci/buildbucket/proto.Build`. The input.properties field of this protobuf follows the `infra/experimental/golangbuildpb.Inputs` schema which represents input parameters that are specific to this luciexe, but may also contain namespaced properties that are injected by different services. For instance, CV uses the "$recipe_engine/cq" namespace.

As an example, to try out a "try bot" path, try the following `build.jsonpb`:

{
	"builder": {
		"project": "go",
		"bucket": "try",
		"builder": "linux-amd64"
	},
	"input": {
		"properties": {
			"project": "go",
			"bootstrap_version": "1.20.6",
			"host": {
				"goarch": "amd64",
				"goos": "linux"
			},
			"target": {
				"goarch": "amd64",
				"goos": "linux"
			},
			"tools_cache": "tools",
			"git_cache": "gitcache",
			"$recipe_engine/cq": {
				"active": true,
				"runMode": "DRY_RUN"
			}
		},
		"gerrit_changes": [
			{
				"host": "go-review.googlesource.com",
				"project": "go",
				"change": 460376,
				"patchset": 1
			}
		]
	}
}

NOTE: by default, a luciexe fake build will discard the temporary directory created to run the build. If you'd like to retain the contents of the directory, specify a working directory to the golangbuild luciexe via the `--working-dir` flag. Be careful about where this working directory lives; particularly, make sure it isn't a subdirectory of a Go module a directory containing a go.mod file.

## Contributing

To keep things organized and consistent, keep to the following guidelines:

• Only functions generate steps. Methods never generate steps.
• Keep step presentation and command execution separate from logic where possible (which will make it easier to write unit tests).
• Steps should be function-scoped. Steps should be created at the start of a function with the step end immediately deferred to function exit.
• Prefer wrapping errors (%w) instead of rendering them (%v). Errors contain rich information like log links that may get lost when rendering them.

## Experiments

When adding new functionality, consider putting it behind an experiment. Experiments are made available in the buildSpec and are propagated from the builder definition. Experiments in the builder definition are given a probability of being enabled on any given build, but always manifest in the current build as either "on" or "off." Experiments should have a name like "golang.my_specific_new_functionality" and should be checked for via spec.experiment("golang.my_specific_new_functionality").

Experiments can be disabled at first (no work needs to be done on the builder definition), rolled out, and then tested in a real build environment via `led`

led get-build ... | \
led edit \
  -experiment golang.my_specific_new_functionality=true | \
led launch

or via `bb add -ex "+golang.my_specific_new_functionality" ...`.

Experiments can be enabled on LUCIEXE_FAKEBUILD runs through the "experiments" property (array of strings) on "input."

### Current experiments

• golang.force_test_outside_repository: Can be used to force running tests from outside the repository to catch accidental reads outside of module boundaries despite the repository not having opted-in to this test behavior.
• golang.shard_by_weight: Use an alternative test sharding strategy that uses precomputed weights to achieve a better balance of work across shards.
• luci.best_effort_platform: set by LUCI on less-supported platforms. Affects which tools are downloaded from CIPD.

## Set up gomote environment locally

If golangbuild is invoked with GOMOTE_SETUP=<project>/<bucket>/<builder>, then the luciexe protocol will be bypassed, and it will set up that builder's tools and environment in the current working directory. Then, it will interpret all command-line arguments as another command to execute.

This is used during gomote setup as a wrapper for the buildlet, but can also be used locally to partially replicate an environment for a builder. For example, the following command will replicate the environment for gotip-linux-amd64 and drop you into a shell:

GOMOTE_SETUP=golang/ci-workers/gotip-linux-amd64-test_only golangbuild bash