go/

README

Go SDK

Usage

TODO(jsharpe): Write usage instructions.

remotetool

remotetool is a CLI tool to interact with remote build execution. In particular, blobs and input trees can be downloaded and uploaded from CAS, actions and their inputs can be downloaded to be either re-run locally or remotely.

Common flags

Flags that describe the RBE instance used and the authentication method are common to most remotetool invocations. The --instance, --service, and credential flags (--use_application_default_credentials, --use_gce_credentials, --credential_file=FILE, one of which must be set), in particular, are further documented under pkg/flags/flags.go.

--alsologtostderr and -v VERBOSITY_LEVEL can be used to tune the verbosity and location of the output, e.g. --alsologtosrderr -v 1.

For example, the common flags could be:

    --service remotebuildexecution.googleapis.com:443 \
    --use_application_default_credentials=true \
    --alsologtostderr \
    --v 1

Downloading an action's inputs and metadata

bazelisk run //go/cmd/remotetool -- \
  --operation=download_action \
  COMMON_FLAGS \
  --digest=DIGEST \
  --path=PATH

• For COMMON_FLAGS, see the above section
• The --digest flag is formatted as "{hash}/{size}" and can typically be obtained from logs of the tool that originally ran the action.
• The --path flag is the destination where the action inputs will be downloaded (under PATH/input), as well as:
  • metadata relating to the action (e.g. command hash, input tree hash, individual input file hashes...)
  • run_command.sh, a script that contains the command line of the action to be re-run
  • run_locally.sh, a script that contains the docker invocation to re-run the action locally

The action can then be re-run locally by running ./run_locally.sh, or remotely using the instructions below. When running the action locally, local modifications made to the input folder will be picked up automatically, but not if re-running the action remotely, read on for more on this.

Re-running a downloaded action remotely

bazelisk run //go/cmd/remotetool -- \
  --operation=execute_action \
  COMMON_FLAGS
  [--action_root=PATH|--digest=DIGEST] \
  --path OUTPUT_PATH

• For COMMON_FLAGS, see the above section.
• An action digest formatted as "{hash}/{size}" can be provided directly using the --digest flag, or a previously downloaded action can be used with --action_root. The --action_root path should point to the --path of a previous download_action invocation. Specifically, the --action_root folder must contain a cmd.textproto and ac.textproto files.
• --path is the destination where the outputs of the action will be downloaded

Re-running a downloaded action remotely on a different RBE instance

If you want to download an action from an RBE instance FOO, and remotely execute it on another RBE instance BAR (probably because you only have the permission to download actions from FOO, but don't have the permission to execute that instance). In this case, you want to download the action first, and then re-run it on another RBE instance with --instance and --action_root flags.

1. download the action from instance FOO to dir /tmp/out/downloaded_action

bazelisk run //go/cmd/remotetool -- \
    --operation download_action \
    --instance=FOO \
    --digest=DIGEST
    COMMON_FLAGS
    --path /tmp/out/downloaded_action

1. run the downloaded action with another RBE instance BAR

bazelisk run //go/cmd/remotetool -- \
    --operation execute_action \
    --instance=BAR \
    --action_root /tmp/out/downloaded_action \
    COMMON_FLAGS
    --path /tmp/output_file_path

Running a modified version of a downloaded action

1. Download an action to a given PATH following the instructions for downloading an action,
2. modify the input under PATH/input as needed,
3. run the action and be sure to include the --action_root=PATH argument.

Alternatively, you could also:

1. upload the newly formed input directory using remotetool:

   bazelisk run //go/cmd/remotetool -- \
     --operation=upload_dir \
     COMMON_FLAGS \
     --path=PATH/input
   

   remotetool will log the digest of the newly uploaded input folder, NEW_DIGEST.
2. In the ac.textproto file under PATH, update the input_root_digest fields with the NEW_DIGEST information.
3. Run the action using the instructions for re-running a downloaded action

Development

Update BUILD.bazel files with (with fix to be more aggressive):

bazel run //:gazelle [fix]

Update go.mod files with:

go mod tidy

Format all files with:

gofmt -w go

Check your code for lint/vet warnings (install golint with sudo apt-get install golint):

golint ./...
go vet ./...

Run all tests:

bazel test ...

or

go test ./...