scheduling

Documentation

Index

• Constants
• Variables
• func ComputeBlamelist(ctx context.Context, cache cache.TaskCache, repo *repograph.Graph, ...) ([]string, *types.Task, error)
• type TaskCandidateSearchTerms
• type TaskScheduler
  • func NewTaskScheduler(ctx context.Context, d db.DB, bl *blacklist.Blacklist, period time.Duration, ...) (*TaskScheduler, error)
  • func (s *TaskScheduler) AddTasks(ctx context.Context, taskMap map[string]map[string][]*types.Task) error
  • func (s *TaskScheduler) CancelJob(id string) (*types.Job, error)
  • func (ts *TaskScheduler) GetBlacklist() *blacklist.Blacklist
  • func (s *TaskScheduler) GetJob(id string) (*types.Job, error)
  • func (s *TaskScheduler) GetTask(id string) (*types.Task, error)
  • func (s *TaskScheduler) HandleSwarmingPubSub(msg *swarming.PubSubTaskMessage) bool
  • func (s *TaskScheduler) MainLoop(ctx context.Context) error
  • func (s *TaskScheduler) MaybeTriggerPeriodicJobs(ctx context.Context, triggerName string) error
  • func (s *TaskScheduler) QueueLen() int
  • func (s *TaskScheduler) RecentSpecsAndCommits() ([]string, []string, []string)
  • func (s *TaskScheduler) RecurseAllBranches(ctx context.Context, ...) error
  • func (s *TaskScheduler) SearchQueue(q *TaskCandidateSearchTerms) []*taskCandidate
  • func (s *TaskScheduler) Start(ctx context.Context, enableTryjobs bool, beforeMainLoop func())
  • func (s *TaskScheduler) Status() *TaskSchedulerStatus
  • func (s *TaskScheduler) TriggerJob(ctx context.Context, repo, commit, jobName string) (string, error)
  • func (s *TaskScheduler) ValidateAndAddTask(ctx context.Context, task *types.Task) error
  • func (s *TaskScheduler) ValidateAndUpdateTask(task *types.Task) error
• type TaskSchedulerStatus

Constants

const (
	// Metric name for free bots.
	MEASUREMENT_FREE_BOT_COUNT = "free_bot_count"

	// FILTER_* are used as the value of the "filter" key in metrics; we record counts for all free
	// bots and all free bots after allocating pending tasks to bots.
	FILTER_ALL_FREE_BOTS       = "all_free_bots"
	FILTER_MINUS_PENDING_TASKS = "minus_pending_tasks"
)

const (
	// Manually-forced jobs have high priority.
	CANDIDATE_SCORE_FORCE_RUN = 100.0

	// Try jobs have high priority, equal to building at HEAD when we're
	// 5 commits behind.
	CANDIDATE_SCORE_TRY_JOB = 10.0

	// When retrying a try job task that has failed, prioritize the retry
	// lower than tryjob tasks that haven't run yet.
	CANDIDATE_SCORE_TRY_JOB_RETRY_MULTIPLIER = 0.75

	// MAX_BLAMELIST_COMMITS is the maximum number of commits which are
	// allowed in a task blamelist before we stop tracing commit history.
	MAX_BLAMELIST_COMMITS = 500

	// Measurement name for task candidate counts by dimension set.
	MEASUREMENT_TASK_CANDIDATE_COUNT = "task_candidate_count"

	NUM_TOP_CANDIDATES = 50
)

Variables

var (
	// Don't schedule on these branches.
	// WARNING: Any commit reachable from any of these branches will be
	// skipped. So, for example, if you fork a branch from head of master
	// and immediately blacklist it, no tasks will be scheduled for any
	// commits on master up to the branch point.
	// TODO(borenet): An alternative would be to only follow the first
	// parent for merge commits. That way, we could remove the checks which
	// cause this issue but still blacklist the branch as expected. The
	// downside is that we'll miss commits in the case where we fork a
	// branch, merge it back, and delete the new branch head.
	BRANCH_BLACKLIST = map[string][]string{
		common.REPO_SKIA_INTERNAL: {
			"skia-master",
		},
	}

	ERR_BLAMELIST_DONE = errors.New("ERR_BLAMELIST_DONE")
)

Functions

func ComputeBlamelist

func ComputeBlamelist(ctx context.Context, cache cache.TaskCache, repo *repograph.Graph, taskName, repoName string, revision *repograph.Commit, commitsBuf []*repograph.Commit, newTasks map[types.RepoState]util.StringSet) ([]string, *types.Task, error)

ComputeBlamelist computes the blamelist for a new task, specified by name, repo, and revision. Returns the list of commits covered by the task, and any previous task which part or all of the blamelist was "stolen" from (see below). There are three cases:

1. The new task tests commits which have not yet been tested. Trace commit history, accumulating commits until we find commits which have been tested by previous tasks.

2. The new task runs at the same commit as a previous task. This is a retry, so the entire blamelist of the previous task is "stolen".

3. The new task runs at a commit which is in a previous task's blamelist, but no task has run at the same commit. This is a bisect. Trace commit history, "stealing" commits from the previous task until we find a commit which was covered by a *different* previous task.

Args:

• cache: TaskCache instance.
• repo: repograph.Graph instance corresponding to the repository of the task.
• taskName: Name of the task.
• repoName: Name of the repository for the task.
• revision: Revision at which the task would run.
• commitsBuf: Buffer for use as scratch space.

Types

type TaskCandidateSearchTerms

type TaskCandidateSearchTerms struct {
	types.TaskKey
	Dimensions []string `json:"dimensions"`
}

TaskCandidateSearchTerms includes fields used for searching task candidates.

type TaskScheduler

type TaskScheduler struct {
	// contains filtered or unexported fields
}

TaskScheduler is a struct used for scheduling tasks on bots.

func NewTaskScheduler

func NewTaskScheduler(ctx context.Context, d db.DB, bl *blacklist.Blacklist, period time.Duration, numCommits int, workdir, host string, repos repograph.Map, isolateClient *isolate.Client, swarmingClient swarming.ApiClient, c *http.Client, timeDecayAmt24Hr float64, buildbucketApiUrl, trybotBucket string, projectRepoMapping map[string]string, pools []string, pubsubTopic, depotTools string, gerrit gerrit.GerritInterface, btProject, btInstance string, ts oauth2.TokenSource) (*TaskScheduler, error)

func (*TaskScheduler) AddTasks

func (s *TaskScheduler) AddTasks(ctx context.Context, taskMap map[string]map[string][]*types.Task) error

AddTasks inserts the given tasks into the TaskDB, updating blamelists. The provided Tasks should have all fields initialized except for Commits, which will be overwritten, and optionally Id, which will be assigned if necessary. AddTasks updates existing Tasks' blamelists, if needed. The provided map groups Tasks by repo and TaskSpec name. May return error on partial success. May modify Commits and Id of argument tasks on error.

func (*TaskScheduler) CancelJob

func (s *TaskScheduler) CancelJob(id string) (*types.Job, error)

CancelJob cancels the given Job if it is not already finished.

func (*TaskScheduler) GetBlacklist

func (ts *TaskScheduler) GetBlacklist() *blacklist.Blacklist

func (*TaskScheduler) GetJob

func (s *TaskScheduler) GetJob(id string) (*types.Job, error)

GetJob returns the given Job.

func (*TaskScheduler) GetTask

func (s *TaskScheduler) GetTask(id string) (*types.Task, error)

GetTask returns the given Task.

func (*TaskScheduler) HandleSwarmingPubSub

func (s *TaskScheduler) HandleSwarmingPubSub(msg *swarming.PubSubTaskMessage) bool

HandleSwarmingPubSub loads the given Swarming task ID from Swarming and updates the associated types.Task in the database. Returns a bool indicating whether the pubsub message should be acknowledged.

func (*TaskScheduler) MainLoop

func (s *TaskScheduler) MainLoop(ctx context.Context) error

MainLoop runs a single end-to-end task scheduling loop.

func (*TaskScheduler) MaybeTriggerPeriodicJobs

func (s *TaskScheduler) MaybeTriggerPeriodicJobs(ctx context.Context, triggerName string) error

MaybeTriggerPeriodicJobs triggers all periodic jobs with the given trigger name, if those jobs haven't already been triggered.

func (*TaskScheduler) QueueLen

func (s *TaskScheduler) QueueLen() int

QueueLen returns the length of the queue.

func (*TaskScheduler) RecentSpecsAndCommits

func (s *TaskScheduler) RecentSpecsAndCommits() ([]string, []string, []string)

RecentSpecsAndCommits returns the lists of recent JobSpec names, TaskSpec names and commit hashes.

func (*TaskScheduler) RecurseAllBranches

func (s *TaskScheduler) RecurseAllBranches(ctx context.Context, fn func(string, *repograph.Graph, *repograph.Commit) (bool, error)) error

recurseAllBranches runs the given func on every commit on all branches, with some Task Scheduler-specific exceptions.

func (*TaskScheduler) SearchQueue

func (s *TaskScheduler) SearchQueue(q *TaskCandidateSearchTerms) []*taskCandidate

SearchQueue returns all task candidates in the queue which match the given TaskKey. Any blank fields are considered to be wildcards.

func (*TaskScheduler) Start

func (s *TaskScheduler) Start(ctx context.Context, enableTryjobs bool, beforeMainLoop func())

Start initiates the TaskScheduler's goroutines for scheduling tasks. beforeMainLoop will be run before each scheduling iteration.

func (*TaskScheduler) Status

func (s *TaskScheduler) Status() *TaskSchedulerStatus

Status returns the current status of the TaskScheduler.

func (*TaskScheduler) TriggerJob

func (s *TaskScheduler) TriggerJob(ctx context.Context, repo, commit, jobName string) (string, error)

TriggerJob adds the given Job to the database and returns its ID.

func (*TaskScheduler) ValidateAndAddTask

func (s *TaskScheduler) ValidateAndAddTask(ctx context.Context, task *types.Task) error

ValidateAndAddTask inserts the given task into the TaskDB, updating blamelists. Checks that the task has a valid repo, revision, name, etc. The task should have all fields initialized except for Commits and Id, which must be empty. Updates existing Tasks' blamelists, if needed. May modify Commits and Id on error.

func (*TaskScheduler) ValidateAndUpdateTask

func (s *TaskScheduler) ValidateAndUpdateTask(task *types.Task) error

ValidateAndUpdateTask modifies the given task in the TaskDB. Ensures the task's blamelist, repo, revision, etc. do not change. The task should have all fields initialized.

type TaskSchedulerStatus

type TaskSchedulerStatus struct {
	LastScheduled time.Time        `json:"last_scheduled"`
	TopCandidates []*taskCandidate `json:"top_candidates"`
}

TaskSchedulerStatus is a struct which provides status information about the TaskScheduler.