recipe_wrapper

README

Recipe Wrapping

The recipe wrapping tool reads a build.proto from stdin, resolves the appropriate recipe version and properties to use, and executes the build.

The tool implements the luciexe protocol. Fuchsia Buildbucket builders which require recipe versioning should configure the tool as their executable. See the lucicfg documentation for more information.

Behavior

Builder properties resolution

recipe_wrapper will resolve the build's input properties by reading properties from a file in integration.git, at a revision determined by the input build.proto.

The path to the properties file is calculated based on the builder information in the input build.proto, specifically infra/config/generated/<project>/properties/<bucket>/<builder>.json, where <project>, <bucket>, and <builder> each come from the corresponding field of the input build.proto under the top-level builder field.

This makes it possible to "version" builder properties – i.e. property changes can be tested in CQ, and properties don't need to be kept backwards compatible with old versions of code on release branches.

However, some properties cannot be versioned because they're needed by recipe_wrapper to resolve the integration repository and revision to checkout. See the "Properties" section for a full listing.

Recipe version resolution

The "recipe version" is a recipes.git revision, which is resolved through a decision tree on build.proto. See main file.

The recipe version is normally read from a file in the same integration.git checkout that the properties file comes from. However, if the build input contains a change to the recipes repository, recipe_wrapper will check out the recipes repository at the input change, rather than at the version pinned in integration. That allows us to test recipe changes in presubmit using regular tryjobs.

Pinning a recipe version in integration.git makes it possible to "version" recipes by using correspondingly old versions of recipes on old release branches. That way, HEAD of recipes does not need to maintain backwards compatibility with every integration.git release branch we wish to build and test, making recipe development much easier.

Recipe checkout & execution

After recipe_wrapper has resolved a recipe version, is checks out recipes.git at the resolved revision and execs recipes.py luciexe within the recipes checkout to take over the remainder of the build execution.

Properties

recipe_wrapper reads several optional builder properties to determine its behavior:

recipe_integration_remote

Example: "https://fuchsia-internal.googlesource.com/integration"

If a build is triggered by a cron schedule rather than by a commit (via LUCI Scheduler) or CL (via Commit Queue), then it won't have a gitiles_commit or gerrit_change in its input.

For public builders it's safe to fall back to using the public integration repository. However, many internal builders assume access to the internal integration repository, so they need to fall back to using internal integration instead.

recipe_integration_remote lets internal builders that may be triggered on a cron job fall back to checking out the specified integration repository (generally the internal integration repository) rather than the default.

recipes_host_override

Example: "fuchsia.googlesource.com"

Some builders are triggered by commits in Git-on-Borg repositories that are not on one of the Fuchsia-owned Git-on-Borg hosts. Only Fuchsia-owned hosts have integration repositories, so for these builds it's not valid to try to checkout the integration repository that's on the same host and ref as the triggering commit.

To avoid that, a builder that's triggered by a non-Fuchsia-owned repo can set recipes_host_override to hostname of the Fuchsia-owned Git-on-Borg instance that holds the desired integration repository. If recipes_host_override is set, we use the refs/heads/main ref for checkout because there's no guarantee for non-Fuchsia-owned repositories that the triggering ref will correspond to a ref of the integration repository.

recipes_integration_project_override

Example: "alternate-integration"

Some builders may checkout an alternate, integration-like Git-on-Borg project which nevertheless contains valid recipe versioning and property files. This property overrides the default "integration" project with the given project for resolution of said files.

recipes_integration_ref_override

Example: "refs/heads/main"

This property overrides the integration.git ref that recipe_wrapper resolves in the case where a resolved integration base revision isn't already specified in the build input.

Similar to recipes_host_override, this is useful for builders that are triggered by non-Fuchsia-owned repositories, for which we can't assume that the triggering ref will correspond to a valid integration.git ref.

recipes_integration_ref_mapping

Example: {"refs/heads/foo": "refs/heads/bar"}

This property also overrides the integration.git ref that recipe_wrapper resolves, but only depending on the original resolved ref (the ref from the build input). Given the above example property value, if the build is triggered on a change to the foo branch, recipe_wrapper would check out the bar branch of the integration repository.

This is useful for builders that checkout integration repositories that use different branch naming schemes from global integration, and that still need to run against changes to global integration-based repositories.

Testing with led

recipe_wrapper has unit tests, and in general as much logic as possible should be covered by unit tests. However, because it has a large interface with Buildbucket and depends on services like Git-on-Borg, unit tests aren't sufficient to guarantee correctness of recipe_wrapper changes.

The easiest way to integration-test recipe_wrapper changes is with led. Testing recipe_wrapper changes with led is slightly more involved than testing recipe changes, because by default led jobs run recipes.py directly instead of using recipe_wrapper.

To circumvent this behavior, you need to build recipe_wrapper locally and upload the resulting executable to a file with the special name luciexe in the led job's isolated input. This work is handled by scripts/led-edit-recipe-wrapper.py, which can be pipelined with other led commands. For example:

led get-build -real-build 8841535313446777120 | ./scripts/led-edit-recipe-wrapper.py | led launch