backends

package
v0.0.0-...-2ad74da Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 21, 2024 License: BSD-3-Clause Imports: 34 Imported by: 0

Documentation

Overview

Package backends provides common logic to provide connections to other backend services that bisection relies on

Index

Constants

View Source 
const (
	// RBE CAS isolates expire after 32 days. We use 30 out of caution.
	CasExpiration = 30
	// ChromeProject refers to the "chrome" project.
	ChromeProject = "chrome"
	// ChromiumGitilesURL is the default Gitiles URL for chromium/src.
	ChromiumGitilesURL = "https://chromium.googlesource.com/chromium/src"
	// ChromiumGitilesHost is the default Gitiles host for chromium/src.
	ChromiumGitilesHost = "chromium.googlesource.com"
	// ChromiumGitilesProject is the default project name for chromium/src.
	ChromiumGitilesProject = "chromium/src"
	// ChromiumGitilesRefAtHead is the default ref used for Chromium builds.
	ChromiumGitilesRefAtHead = "refs/heads/main"
	// DefaultBucket is the Pinpoint bucket, equivalent to the "try" builds in Buildbucket.
	DefaultBucket = "try"
	// DefaultBuildsetKey is key tagged on builds for how commit information is tracked in Waterfall (CI) and Pinpoint.
	DefaultBuildsetKey = "buildset"
	// DefaultTagValue is the value format for the key above.
	DefaultBuildsetValue = "commit/gitiles/chromium.googlesource.com/chromium/src/+/%s"
	// DefaultCASInstance is the default CAS instance used by Pinpoint builds.
	//
	// TODO(b/315215756): Support other swarming instances. There are three known
	// swarming instances Pinpoint supports. The majority of Pinpoint builds are
	// this defaultInstance. Buildbucket API does not report the swarming instance
	// so our options are to:
	// - include the expected instance in the build tags
	// - try all 3 known swarming instances and brute force it
	DefaultCASInstance = "projects/chrome-swarming/instances/default_instance"
	// DefaultPerRPCTimeout defines the default time permitted for each RPC call.
	DefaultPerRPCTimeout = 90 * time.Second
	// DefaultRetries is the default number of retries for Backoff logic to Buildbucket.
	DefaultRetries = 10
	// DepsOverrideKey is the key used to find any deps overrides in the input properties from a Buildbucket response.
	DepsOverrideKey = "deps_revision_overrides"
	// SwarmingHashRefKey is the key used to find CAS hashes from successful Pinpoint Buildbucket builds.
	SwarmingHashRefKey = "swarm_hashes_refs"
	// WaterfallBucket is equivalent to the "ci" bucket in Buildbucket.
	WaterfallBucket = "ci"
)
View Source 
const (
	DefaultSwarmingServiceAddress = "chrome-swarming.appspot.com:443"
	RunBenchmarkFailure           = "BENCHMARK_FAILURE" // the task completed but benchmark run failed
)
View Source 
const (
	ChromiumRepo = "chromium/src"
)
View Source 
const (
	DEFAULT_GITILES_SCOPE = "https://www.googleapis.com/auth/gerritcodereview"
)

Variables

View Source 
var PinpointWaterfall = map[string]string{
	"Android Compile Perf":                       "android-builder-perf",
	"Android Compile Perf PGO":                   "android-builder-perf-pgo",
	"Android arm64 Compile Perf":                 "android_arm64-builder-perf",
	"Android arm64 Compile Perf PGO":             "android_arm64-builder-perf-pgo",
	"Android arm64 High End Compile Perf":        "android_arm64_high_end-builder-perf",
	"Android arm64 High End Compile Perf PGO":    "android_arm64_high_end-builder-perf-pgo",
	"Chromecast Linux Builder Perf":              "chromecast-linux-builder-perf",
	"Chromeos Amd64 Generic Lacros Builder Perf": "chromeos-amd64-generic-lacros-builder-perf",
	"Fuchsia Builder Perf":                       "fuchsia-builder-perf-arm64",
	"Linux Builder Perf":                         "linux-builder-perf",
	"Linux Builder Perf PGO":                     "linux-builder-perf-pgo",
	"Mac Builder Perf":                           "mac-builder-perf",
	"Mac Builder Perf PGO":                       "mac-builder-perf-pgo",
	"Mac arm Builder Perf":                       "mac-arm-builder-perf",
	"Mac arm Builder Perf PGO":                   "mac-arm-builder-perf-pgo",
	"mac-laptop_high_end-perf":                   "mac-laptop_high_end-perf",
	"Win x64 Builder Perf":                       "win64-builder-perf",
	"Win x64 Builder Perf PGO":                   "win64-builder-perf-pgo",
}

Builds from waterfall can also be recycled for bisection

As part of our anomaly detection system, Waterfall builders will continuously build Chrome near the tip of main. Sometimes, Pinpoint jobs will attempt to build a CL Waterfall has already built i.e. verifying a regression. These builds are automatic. Pinpoint builders will only build Chrome on demand. The Waterfall and Pinpoint builders are maintained in separate pools.

The map is maintained here: https://chromium.googlesource.com/chromium/tools/build/+/986f23767a01508ad1eb39194ffdb5fec4f00d7b/recipes/recipes/pinpoint/builder.py#22 TODO(b/316207255): move this builder map to a more stable config file

Functions

func CreateRepository 

func CreateRepository(ctx context.Context, repositoryUrl string) (*gitiles.Repo, error)

CreateRepository returns an authenticated client to communicate with Gitiles.

func NewBigQueryClient 

func NewBigQueryClient(ctx context.Context, project string) (*bigQueryClient, error)

NewBigQueryClient returns a BigQueryClient for the provided project.

func NewBuildbucketClient 

func NewBuildbucketClient(bc bpb.BuildsClient) *buildbucketClient

func NewIssueTrackerTransport 

func NewIssueTrackerTransport(ctx context.Context) (*issueTrackerTransport, error)

NewIssueTrackerTransport returns a issueTrackerTransport object configured with templates.

Types

type BigQueryClient 

type BigQueryClient interface {
	// CreateTable creates the dataset and table for the project that the client was instantiated with.
	// schema is expected to be a struct with bigquery tags defined for column definitions.
	CreateTable(ctx context.Context, datasetID, tableName string, schema interface{}) error

	// Insert implements BigQuery PUT.
	Insert(ctx context.Context, datasetID, tableName string, rows interface{}) error
}

BigQueryClient interfaces interactions to BigQuery

type BuildbucketClient 

type BuildbucketClient interface {
	// CancelBuild sends a cancellation request to Buildbucket. It's expected that
	// Buildbucket will cancel the build, whether that's graceful termination or
	// forced cancellation, as long as the request is received.
	CancelBuild(ctx context.Context, buildID int64, summary string) error

	// GetSingleBuild calls Buildbucket to find existing builds for the
	// given builder, Chromium revision, DEPS overrides and Gerrit patches.
	//
	// TODO(b/315215756): The current mechanism can be updated to utilize
	// tags, so that we aren't operating on O(len(builds) * len(deps_overrides))
	// to find the exact builds. This will require tagging scheduled builds with
	// new tags before it can be utilized.
	GetSingleBuild(ctx context.Context, builderName, bucket, commit string, deps map[string]string, patches []*bpb.GerritChange) (*bpb.Build, error)

	// GetBuildWithDeps search for a build with matching deps.
	// Overloaded method of GetSingleBuild().
	GetBuildWithDeps(ctx context.Context, builderName, bucket, commit string, deps map[string]string) (*bpb.Build, error)

	// GetBuildWithPatches searches for a build with matching patches.
	// Overloaded method of GetSingleBuild().
	GetBuildWithPatches(ctx context.Context, builderName, bucket, commit string, patches []*bpb.GerritChange) (*bpb.Build, error)

	// GetBuildFromWaterfall searches for an existing build using its waterfall
	// (CI) counterpart.
	GetBuildFromWaterfall(ctx context.Context, builderName, commit string) (*bpb.Build, error)

	// GetBuildStatus returns the build status given the ID.
	GetBuildStatus(ctx context.Context, buildID int64) (bpb.Status, error)

	// GetCASReference returns a CAS reference to the output artifacts of a successful build.
	GetCASReference(ctx context.Context, buildID int64, target string) (*apipb.CASReference, error)

	// StartChromeBuild triggers a Chrome build.
	StartChromeBuild(ctx context.Context, pinpointJobID, requestID, builderName, commitHash string, deps map[string]string, patches []*bpb.GerritChange) (*bpb.Build, error)

	// StartBuild triggers a Buildbucket request.
	StartBuild(ctx context.Context, req *bpb.ScheduleBuildRequest) (*bpb.Build, error)
}

type BuildbucketClientConfig 

type BuildbucketClientConfig struct {
	// The buildbucket host to target. See "go.skia.org/infra/go/buildbucket"
	// for the default value.
	Host string

	// Retries, if >= 0, is the number of remaining retries. If <0, no retry
	// count will be applied.
	Retries int

	// Delay is the next generated delay.
	Delay time.Duration

	// MaxDelay is the maximum duration. If <= zero, no maximum will be enforced.
	MaxDelay time.Duration

	// PerRPCTimeout, if > 0, is a timeout that is applied to each call attempt.
	PerRPCTimeout time.Duration
}

BuildbucketClientConfig represents options for the behavior of the Buildbucket client.

Example: bc := DefaultClientConfig().WithClient(c) bc.GetSingleBuild(...)

func DefaultClientConfig 

func DefaultClientConfig() BuildbucketClientConfig

DefaultClientConfig returns a BuildbucketClientConfig with defaults:

  • Host: cr-buildbucket.appspot.com
  • Exponential backoff with 10 retries
  • PerRPCTimeout of 90 seconds. Swarming servers have an internal 60-second deadline to respond to requests.

func (BuildbucketClientConfig) WithClient 

func (bc BuildbucketClientConfig) WithClient(c *http.Client) *buildbucketClient

WithClient returns a BuildbucketClient as configured by the ClientConfig

type CrrevClient 

type CrrevClient interface {
	// GetCommitInfo returns the git hash, project, and repo of a commit
	// Supports commit hashes and positions and can return non-chromium commits.
	GetCommitInfo(ctx context.Context, commit string) (*CrrevResponse, error)
}

CrrevClient creates an API for alert grouping and Pinpoint UI to convert commit positions to git hashes before submitting them as Pinpoint jobs.

type CrrevClientImpl 

type CrrevClientImpl struct {
	Client *http.Client
}

CrrevClientImpl implements CrrevClient

func NewCrrevClient 

func NewCrrevClient(ctx context.Context) (*CrrevClientImpl, error)

func (*CrrevClientImpl) GetCommitInfo 

func (c *CrrevClientImpl) GetCommitInfo(ctx context.Context, commit string) (*CrrevResponse, error)

type CrrevResponse 

type CrrevResponse struct {
	GitHash string `json:"git_sha"`
	Project string `json:"project"`
	Repo    string `json:"repo"`
}

CrrevResponse is the response provided by the crrev redirect endpoint

type IssueTracker 

type IssueTracker interface {
	ReportCulprit(issueID int64, culprits []*pinpoint_proto.CombinedCommit) error
}

type SwarmingClient 

type SwarmingClient interface {
	// CancelTasks tells Swarming to cancel the given tasks.
	CancelTasks(ctx context.Context, taskIDs []string) error

	// GetCASOutput returns the CAS output of a swarming task.
	GetCASOutput(ctx context.Context, taskID string) (*apipb.CASReference, error)

	// GetStartTime returns the starting time of the swarming task.
	GetStartTime(ctx context.Context, taskID string) (*timestamppb.Timestamp, error)

	// GetStatus gets the current status of a swarming task.
	GetStatus(ctx context.Context, taskID string) (string, error)

	// TriggerTask is a wrapper around swarming.ApiClient TriggerTask
	TriggerTask(ctx context.Context, req *apipb.NewTaskRequest) (*apipb.TaskRequestMetadataResponse, error)

	// FetchFreeBots gets a list of available bots per specified builder configuration.
	FetchFreeBots(ctx context.Context, builder string) ([]*apipb.BotInfo, error)

	// GetBotTasksBetweenTwoTasks generates a list of tasks that started in between two tasks.
	// This function is primarily used by Pairwise jobs to assess if another swarming task
	// executed in between a pair of tasks.
	GetBotTasksBetweenTwoTasks(ctx context.Context, botID, taskID1, taskID2 string) (*apipb.TaskListResponse, error)
}

SwarmingClient

type SwarmingClientImpl 

type SwarmingClientImpl struct {
	swarmingv2.SwarmingV2Client
}

SwarmingClientImpl TODO(jeffyoon@) make this private once run_benchmark doesn't rely on this in testing.

func NewSwarmingClient 

func NewSwarmingClient(ctx context.Context, server string) (*SwarmingClientImpl, error)

func (*SwarmingClientImpl) CancelTasks 

func (s *SwarmingClientImpl) CancelTasks(ctx context.Context, taskIDs []string) error

CancelTasks tells Swarming to cancel the given tasks.

func (*SwarmingClientImpl) FetchFreeBots 

func (s *SwarmingClientImpl) FetchFreeBots(ctx context.Context, builder string) ([]*apipb.BotInfo, error)

FetchFreeBots gets a list of available bots per specified builder configuration.

func (*SwarmingClientImpl) GetBotTasksBetweenTwoTasks 

func (s *SwarmingClientImpl) GetBotTasksBetweenTwoTasks(ctx context.Context, botID, taskID1, taskID2 string) (*apipb.TaskListResponse, error)

GetBotTasksBetweenTwoTasks generates a list of tasks that started in between two tasks.

func (*SwarmingClientImpl) GetCASOutput 

func (s *SwarmingClientImpl) GetCASOutput(ctx context.Context, taskID string) (*apipb.CASReference, error)

GetCASOutput returns the CAS output of a swarming task in the form of a RBE CAS hash. This function assumes the task is finished, or it throws an error.

func (*SwarmingClientImpl) GetStartTime 

func (s *SwarmingClientImpl) GetStartTime(ctx context.Context, taskID string) (*timestamppb.Timestamp, error)

GetStartTime returns the starting time of the swarming task.

func (*SwarmingClientImpl) GetStatus 

func (s *SwarmingClientImpl) GetStatus(ctx context.Context, taskID string) (string, error)

GetStatus gets the current status of a swarming task.

func (*SwarmingClientImpl) TriggerTask 

func (s *SwarmingClientImpl) TriggerTask(ctx context.Context, req *apipb.NewTaskRequest) (*apipb.TaskRequestMetadataResponse, error)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.