runner

package
v0.0.1-alpha Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 28, 2022 License: Apache-2.0 Imports: 8 Imported by: 0

README

Runner Design

package runner is structs and interfaces to run Commands in Scoot. package runners contains implementations. This doc describes both.

Summary

  • Service (interface) combines Controller and StatusReader
  • Controller (interface) controls when to Run a Command.
    • Controller calls Invoker
    • Controller writes to StatusWriter
  • runners.Invoker (struct) runs one command using Execer and snapshot.Filer
    • implements Abort and Timeout
  • runners.Statuses (struct) offers read and write of RunStatus
    • satisfies interfaces StatusWriter and StatusReader

Invoker (runner/runners/invoker.go)

Invoker runs a Scoot command. This involves:

  • setup (check out a snapshot)
  • post-processing (save output)
  • babysitting (implement timeout and abort)
  • streaming RunStatus'es over the course of a run

Conceptually, it lowers a Scoot Command from a high-level abstraction (Snapshots, etc.) to an Execer command (about directories that are paths in the local filesystem). Concretely, it checks out the input snapshot's files, then uses an Execer to run the command. When the command has finished, it created a snapshot of the command's output.

Controller (runner/controller.go)

Controller controls the Runs, starting a new one (Run()) or ending one (Abort()). It implements:

  • tenancy (run one at a time? Or multiple? Fixed, or based on system utilization?)
  • queueing (should we accept work we can't start right now)

When a Controller wants to start a Run of a Command, it will use an Invoker.

For now we have one implementation, QueueController

Status interfaces (runner/statuses.go)

There are several interfaces having to do with reading/writing RunStatus'es.

StatusWriter for writing Statuses.

There are more reader interfaces:

  • StatusReader includes all the ones below
  • LegacyStatusReader offers legacy APIs
  • StatusQuerier offers Query, which allows waiting for some period of time
  • StatusQueryNower offers QueryNow which is the same as Query, but does not allow waiting

StatusManager implementations (runner/runners/statuses.go)

StatusManager holds RunStatus'es and implements the various Status interfaces.

Service (runner/runner.go)

Service includes the ability to control commands (Controller) and watch the status (StatusReader).

Relation to Execer

Execer is lower-level than runner. Runner deals with Scoot Abstractions; but Execer is just about Unix abstractions: files, directories, etc.

OSExecer calls os/exec.

SimExecer simulates behavior based on the args passed to it. This lets a caller script the behavior of the Execer.

Documentation

Overview

Package runner provides for execution of Scoot work and retrieval of the results of that work. The core of this is the Runner interface. This package includes a standard runner as well as test and simulation implementations.

Index

Constants

View Source 
const (
	UNKNOWN_MASK  StateMask = StateMask(1 << uint(PENDING))
	PENDING_MASK            = 1 << uint(UNKNOWN)
	RUNNING_MASK            = 1 << uint(RUNNING)
	COMPLETE_MASK           = 1 << uint(COMPLETE)
	FAILED_MASK             = 1 << uint(FAILED)
	ABORTED_MASK            = 1 << uint(ABORTED)
	TIMEDOUT_MASK           = 1 << uint(TIMEDOUT)
	DONE_MASK               = (1<<uint(COMPLETE) |
		1<<uint(FAILED) |
		1<<uint(ABORTED) |
		1<<uint(TIMEDOUT) |
		1<<uint(UNKNOWN))

	ALL_MASK = math.MaxUint64
)

Useful StateMask constants

View Source 
const LoggingErrMsg = "Error initializing logging."
View Source 
const NoRunnersMsg = "No runners available."
View Source 
const RunnerBusyMsg = "Runner is busy"

Variables

This section is empty.

Functions

func FinalStatus 

func FinalStatus(q StatusQuerier, id RunID) (RunStatus, ServiceStatus, error)

func SingleStatus 

func SingleStatus(statuses []RunStatus, service ServiceStatus, err error) (RunStatus, ServiceStatus, error)

func StatusAll 

func StatusAll(q StatusQueryNower) ([]RunStatus, ServiceStatus, error)

StatusAll returns the Current status of all runs

func StatusNow 

func StatusNow(q StatusQueryNower, id RunID) (RunStatus, ServiceStatus, error)

Status returns the current status of id from q.

func WaitForState 

func WaitForState(q StatusQuerier, id RunID, expected RunState) (RunStatus, ServiceStatus, error)

Types

type Command 

type Command struct {
	// Command line to run
	Argv []string

	// Key-value pairs for environment variables
	EnvVars map[string]string

	// Kill command after timeout. Zero value is ignored.
	// Time spent prepping for this command (ex: git checkout) is counted towards the timeout.
	Timeout time.Duration

	// Runner can optionally use this to run against a particular snapshot. Empty value is ignored.
	SnapshotID string

	// Runner is given JobID, TaskID, and Tag to help trace tasks throughout their lifecycle
	tags.LogTags

	// Bazel ExecuteRequest data for tasks initiated from the Bazel API
	ExecuteRequest *bazelapi.ExecuteRequest
}

A command, execution environment, and timeout.

func (Command) String 

func (c Command) String() string

type Controller 

type Controller interface {
	// Run prepares cmd. It returns its status and any errors.
	// Run may:
	// enqueue cmd (leading to state PENDING)
	// run cmd immediately (leading to state RUNNING)
	// check if cmd is well-formed, and reject it if not (leading to state FAILED)
	// wait a very short period of time for cmd to start (leading to any state)
	// Run may not wait indefinitely for cmd to finish. This is an async API.
	Run(cmd *Command) (RunStatus, error)

	// Abort kills the given run. Returns its final status (which may not be aborted,
	// e.g. if it's already completed on its own)
	Abort(run RunID) (RunStatus, error)

	// Optional function to clean up process-local resources, ex: cancel goroutines, close connections, etc
	Release()
}

type LegacyStatusReader 

type LegacyStatusReader interface {
	// Status returns the current status of id from q.
	Status(run RunID) (RunStatus, ServiceStatus, error)

	// StatusAll returns the Current status of all runs
	StatusAll() ([]RunStatus, ServiceStatus, error)
}

LegacyStatusReader contains legacy methods to read Status'es. Prefer using the convenience methods above.

type Output 

type Output interface {
	// Write (and close) straight to the Output
	io.WriteCloser

	// A URI to this Output. Clients can read data by accessing the URI.
	// This lets us change how we save, because we can write to a local file,
	// or hdfs, or s3, or any other addressable store.
	// Returns a resource path prefixed with, for example, "file://" or "hdfs://".
	URI() string

	// Copies (if necessary) the URI target to local file and returns that absolute file path.
	AsFile() string
}

Output is a sink for one file's worth of output

type OutputCreator 

type OutputCreator interface {
	// Create an output for the given ID
	Create(id string) (Output, error)
}

OutputCreator lets clients create new Outputs so they can save data. This is how Runner can save stdout and stderr. OutputCreator is the filesystem that creates many Outputs; Output is one file in that. (NB: Saver is temporary until we save output into a new Snapshot)

type Query 

type Query struct {
	Runs    []RunID   // Runs to query for
	AllRuns bool      // Whether to match all runs
	States  StateMask // What States to match
}

Query describes a query for RunStatuses. The Runs and States are and'ed: a RunStatus matches a Query if its ID is in q.Runs (or q.AllRuns) and its state is in q.States

func (Query) Matches 

func (q Query) Matches(st RunStatus) bool

Matches checks if st matches q

type RunID 

type RunID string

type RunState 

type RunState int
const (
	// Catch-all for indeterminate RunStates. Also an "end state"
	UNKNOWN RunState = iota

	// Waiting to run
	PENDING

	// Running
	RUNNING

	// Succeeded or failed yielding an exit code
	COMPLETE

	// Run mechanism failed and is no longer running
	FAILED

	// User requested that the run be killed, or task preempted by Scheduler
	ABORTED

	// Run timed out and was killed
	TIMEDOUT
)

func (RunState) IsDone 

func (p RunState) IsDone() bool

func (RunState) String 

func (p RunState) String() string

type RunStatus 

type RunStatus struct {
	RunID RunID
	State RunState
	tags.LogTags

	// References to stdout and stderr, not their text
	// Runner impls might not provide valid refs for all States (e.g. failure before creation of refs)
	StdoutRef string
	StderrRef string
	// Only valid if State == COMPLETE
	SnapshotID string
	// Only valid if State == (COMPLETE || FAILED)
	ExitCode errors.ExitCode
	// Only valid if State == (COMPLETE || FAILED || ABORTED)
	Error string
	// Only valid if task is run on a Bazel filer
	ActionResult *bazelapi.ActionResult
}

Returned by the coordinator when a run request is made.

func AbortStatus 

func AbortStatus(runID RunID, tags tags.LogTags) (r RunStatus)

func CompleteStatus 

func CompleteStatus(runID RunID, snapshotID string, exitCode errors.ExitCode, tags tags.LogTags) (r RunStatus)

func FailedStatus 

func FailedStatus(runID RunID, err *errors.ExitCodeError, tags tags.LogTags) (r RunStatus)

func PendingStatus 

func PendingStatus(runID RunID, tags tags.LogTags) (r RunStatus)

func RunningStatus 

func RunningStatus(runID RunID, stdoutRef, stderrRef string, tags tags.LogTags) (r RunStatus)

func TimeoutStatus 

func TimeoutStatus(runID RunID, tags tags.LogTags) (r RunStatus)

func (RunStatus) String 

func (p RunStatus) String() string

type RunType 

type RunType string

RunTypes distinguish execution behavior for Commands. Used only by the Runner and evaluated using SnapshotIDs. 

const (
	RunTypeScoot RunType = "Scoot"
	RunTypeBazel RunType = "Bazel"
)

type RunTypeMap 

type RunTypeMap map[RunType]snapshot.FilerAndInitDoneCh

func MakeRunTypeMap 

func MakeRunTypeMap() RunTypeMap

type RunnerID 

type RunnerID struct {
	ID string
}

Can be used to initialize a Runner with specific identifying information 

var EmptyID RunnerID = RunnerID{ID: ""}

func (RunnerID) String 

func (rID RunnerID) String() string

Return a string representation of a RunnerID

type Service 

type Service interface {
	Controller
	StatusReader
}

Service allows starting/abort'ing runs and checking on their status.

type ServiceStatus 

type ServiceStatus struct {
	Initialized bool
	Error       error
}

This is for overall runner status, just 'initialized' status and error for now.

func (ServiceStatus) String 

func (s ServiceStatus) String() string

type StateMask 

type StateMask uint64

StateMask describes a set of States as a bitmask.

func MaskForState 

func MaskForState(state ...RunState) StateMask

Helper Function to create StateMask that matches exactly state

func (StateMask) Matches 

func (m StateMask) Matches(state RunState) bool

func (StateMask) String 

func (sm StateMask) String() string

type StatusQuerier 

type StatusQuerier interface {
	// Query returns all RunStatus'es matching q, waiting as described by w
	Query(q Query, w Wait) ([]RunStatus, ServiceStatus, error)

	StatusQueryNower
}

StatusQuerier allows reading Status by Query'ing.

type StatusQueryNower 

type StatusQueryNower interface {
	// QueryNow returns all RunStatus'es matching q in their current state
	QueryNow(q Query) ([]RunStatus, ServiceStatus, error)
}

StatusQueryNower allows Query'ing Statuses but with no Waiting This is separate from StatusQuerier because talking to the Worker, e.g., we will be able to QueryNow easily, but because Thrift doesn't like long waits on RPCs, it can't do Query with a decent wait. We want our type system to help protect us from this blowing up at runtime, so the RPC client will implement StatusQueryNower. We will implement a PollingQueuer that wraps a StatusQueryNower and satisfies StatusQuerier.

type StatusReader 

type StatusReader interface {
	StatusQuerier

	// TODO(dbentley): remove
	LegacyStatusReader
}

StatusReader includes both the preferred and the legacy api.

type StatusWriter 

type StatusWriter interface {
	// NewRun creates a new RunID in state PENDING
	NewRun() (RunStatus, error)

	// Update overall service status.
	UpdateService(st ServiceStatus) error

	// Update writes a new status.
	Update(st RunStatus) error
}

StatusWriter allows writing Statuses

type Wait 

type Wait struct {
	// How long to wait for Statuses
	Timeout time.Duration
	AbortCh chan interface{}
}

Wait describes how to Wait. It differs from StatusQuery because StatusQuery describes what RunStatus'es to match, but Timeout of zero means return immediately, no blocking.

func WaitForever 

func WaitForever() Wait

Source Files

View all Source files

Directories

Path Synopsis
execers
os
Package mocks is a generated GoMock package.
 Package mocks is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.