runtime

package
v0.1.20 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 3, 2018 License: MPL-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package runtime contains the generic functionality that an engine and plugins use.

Index

Constants

This section is empty.

Variables

View Source 
var ErrFatalInternalError = errors.New("Encountered a fatal internal error")

ErrFatalInternalError is used to signal that a fatal internal error has been logged and that the worker should gracefully terminate/reset.

Engines and plugins can return any unknown error in-order to trigger the same effect. As the worker will report, log and terminate/reset when it encounters an unknown error. This error is ONLY used when the error has already been reported and logged to both system log and task log.

This is only useful for plugins and engines that wishes to manually handle error reporting.

View Source 
var ErrLogNotClosed = errors.New("Log is still open")

ErrLogNotClosed represents an invalid attempt to extract a log while it is still open.

View Source 
var ErrNonFatalInternalError = errors.New("Encountered a non-fatal internal error")

ErrNonFatalInternalError is used to indicate that the operation failed because of internal error that isn't expected to affect other tasks.

Worker need not worry about logging the error to system log or task log as the engine/plugin which returned this error already reported it, log it and/or deemed the error inconsequential.

Worker should, however, report the task as exception and resolve it with reason 'internal-error'. If the worker gets a lot of these non-fatal internal errors, it may employ a heuristic to decide if it has entered a bad state. For example, worker might reboot if it has seen more than 5 non-fatal internal errors within the span of 15min or 5 tasks.

Functions

func NewTaskContext 

func NewTaskContext(tempLogFile string, task TaskInfo) (*TaskContext, *TaskContextController, error)

NewTaskContext creates a TaskContext and associated TaskContextController

func RenderDocument added in v0.1.0 

func RenderDocument(title string, sections []Section) string

RenderDocument creates a markdown document with given title from a list of sections. Ordering sections alphabetically.

Types

type Environment 

type Environment struct {
	GarbageCollector gc.ResourceTracker
	TemporaryStorage
	webhookserver.WebHookServer // Optional, may be nil if not available
	Monitor
	Worker        Stoppable
	ProvisionerID string
	WorkerType    string
	WorkerGroup   string
	WorkerID      string
}

Environment is a collection of objects that makes up a runtime environment.

This type is intended to be passed by value, and should only contain pointers and interfaces for that reason.

type ErrorArtifact added in v0.0.2 

type ErrorArtifact struct {
	Name    string
	Message string
	Reason  string
	Expires time.Time
}

ErrorArtifact wraps all of the needed fields to upload an error artifact

type ExceptionReason added in v0.0.2 

type ExceptionReason int

An ExceptionReason specifies the reason a task reached an exception state. 

const (
	ReasonNoException ExceptionReason = iota
	ReasonCanceled
	ReasonWorkerShutdown
	ReasonMalformedPayload
	ReasonResourceUnavailable
	ReasonInternalError
	ReasonSuperseded
	ReasonIntermittentTask
)

Reasons why a task can reach an exception state. Implementors should be warned that additional entries may be added in the future.

func (ExceptionReason) String added in v0.1.0 

func (e ExceptionReason) String() string

String returns a string repesentation of the ExceptionReason for use with the taskcluster-queue API.

type LifeCycleTracker added in v0.1.0 

type LifeCycleTracker struct {
	StoppingNow        atomics.Once
	StoppingGracefully atomics.Once
}

LifeCycleTracker implements Stoppable as two atomics.Once that you can wait for, or get a blocking channel from.

func (*LifeCycleTracker) StopGracefully added in v0.1.0 

func (s *LifeCycleTracker) StopGracefully()

StopGracefully does StoppingGracefully

func (*LifeCycleTracker) StopNow added in v0.1.0 

func (s *LifeCycleTracker) StopNow()

StopNow does StoppingNow and StoppingGracefully

type MalformedPayloadError added in v0.0.11 

type MalformedPayloadError struct {
	// contains filtered or unexported fields
}

The MalformedPayloadError error type is used to indicate that some operation failed because of malformed-payload.

For example a string expected to be path contained invalid characters, a required property was missing, or an integer was outside the permitted range.

func IsMalformedPayloadError added in v0.0.11 

func IsMalformedPayloadError(err error) (e *MalformedPayloadError, ok bool)

IsMalformedPayloadError casts error to MalformedPayloadError.

This is mostly because it's hard to remember that error isn't supposed to be cast to *MalformedPayloadError.

func MergeMalformedPayload added in v0.0.11 

func MergeMalformedPayload(errors ...*MalformedPayloadError) *MalformedPayloadError

MergeMalformedPayload merges a list of MalformedPayloadError objects

func NewMalformedPayloadError added in v0.0.11 

func NewMalformedPayloadError(a ...interface{}) *MalformedPayloadError

NewMalformedPayloadError creates a MalformedPayloadError object, please make sure to include a detailed description of the error, preferably using multiple lines and with examples.

These will be printed in the logs and end-users will rely on them to debug their tasks.

func (*MalformedPayloadError) Error added in v0.0.11 

func (e *MalformedPayloadError) Error() string

Error returns the error message and adheres to the Error interface

func (*MalformedPayloadError) Messages added in v0.1.1 

func (e *MalformedPayloadError) Messages() []string

Messages returns a list of messages explaining why the error.

type Monitor added in v0.0.11 

type Monitor interface {
	// Measure values in statsum
	Measure(name string, value ...float64)
	// Increment counters in statsum
	Count(name string, value float64)
	// Measure time of fn in statsum
	Time(name string, fn func())

	// Report error/warning to sentry and write to log, returns incidentId which
	// can be included in task-logs, if relevant.
	ReportError(err error, message ...interface{}) string
	ReportWarning(err error, message ...interface{}) string

	// CapturePanic reports panics to log/sentry and returns incidentID, if any
	CapturePanic(fn func()) (incidentID string)

	// Write log messages to system log
	Debug(...interface{})
	Debugln(...interface{})
	Debugf(string, ...interface{})
	Print(...interface{})
	Println(...interface{})
	Printf(string, ...interface{})
	Info(...interface{})
	Infoln(...interface{})
	Infof(string, ...interface{})
	Warn(...interface{})
	Warnln(...interface{})
	Warnf(string, ...interface{})
	Error(...interface{})
	Errorln(...interface{})
	Errorf(string, ...interface{})
	Panic(...interface{})
	Panicln(...interface{})
	Panicf(string, ...interface{})

	// Create child monitor with given tags (tags don't apply to statsum)
	WithTags(tags map[string]string) Monitor
	WithTag(key, value string) Monitor
	// Create child monitor with given prefix (prefix applies to everything)
	WithPrefix(prefix string) Monitor
}

A Monitor is responsible for collecting logs, stats and error messages.

A monitor is a context aware object for monitoring. That is to say that a Monitor is used to record metrics, write logs and report errors. When doing so the Monitor object adds meta-data to the metrics, logs and errors. The meta-data added is context dependent tags and prefix. These help identify where a log message, metric or error originates from.

By encapsulating the context meta-data inside the Monitor object, an implementor gets a Monitor rarely needs to add tags or prefix. For example a monitor will always be prefixed by plugin name before being passed to a plugin, hence, it is easy trace any log message, metric or error report to the plugin that it was created in.

When passing a Monitor to a sub-component it often makes sense to add additional tags or prefix. This way a downloader function that takes a Monitor need not worry about being able to distinguish its metrics, logs and errors from that of its parent.

Prefixes should always be constants, such as engine, plugin, function or component names. Values that change such as taskId or runId should not be used as prefixes, such values is however great as tags.

All metrics reported for a given prefix + name will be aggregated. Hence, if taskId was used as prefix, the dimensionality of metrics would explode and the aggregation wouldn't be useful.

type RedirectArtifact added in v0.0.2 

type RedirectArtifact struct {
	Name     string
	Mimetype string
	URL      string
	Expires  time.Time
}

RedirectArtifact wraps all of the needed fields to upload a redirect artifact

type S3Artifact added in v0.0.2 

type S3Artifact struct {
	Name              string
	Mimetype          string
	Expires           time.Time
	Stream            ioext.ReadSeekCloser
	AdditionalHeaders map[string]string
}

S3Artifact wraps all of the needed fields to upload an s3 artifact

type Section added in v0.1.0 

type Section struct {
	Title   string // Title of section rendered as headline level 2
	Content string // Section contents, maybe contains headline level 3 and higher
}

A Section represents a section of markdown documentation.

type ShutdownManager added in v0.0.2 

type ShutdownManager interface {
	WaitForShutdown() <-chan struct{}
}

ShutdownManager implements a method for listening for shutdown events. Consumers

func NewShutdownManager added in v0.0.2 

func NewShutdownManager(host string) ShutdownManager

NewShutdownManager will return a shutdown manager appropriate for the host that the worker is being run on.

Shutdown events are triggered different ways depending on where the worker is running. When running in AWS, then notifications are sent on their meta-data api, but running locally could cause the worker to represent to different kind of shutdown events.

type Stoppable added in v0.1.0 

type Stoppable interface {
	// StopNow causes the worker to stop processing tasks, resolving all active
	// tasks exception w. worker-shutdown.
	StopNow()
	// StopGracefully causes the worker to stop claiming tasks and stop gracefully
	// when all active tasks have been resolved.
	StopGracefully()
}

Stoppable is an worker with a life-cycle that can be can be stopped.

type StoppableOnce added in v0.1.0 

type StoppableOnce struct {
	Stoppable Stoppable
	// contains filtered or unexported fields
}

StoppableOnce is a wrapper that ensures we only call StopGracefully and StopNow once and never call StopGracefully after StopNow.

There is never any harm in wrapping with this, it merely limits excessive calls to StopNow() and StopGracefully(). Please note that Stoppable.StopNow() may still be invoked after Stoppable.StopGracefully(), it can even be invoked concurrently.

func (*StoppableOnce) StopGracefully added in v0.1.0 

func (s *StoppableOnce) StopGracefully()

StopGracefully calls StopGracefully() on the s.Stoppable, if neither StopGracefully() or StopNow() have been called.

func (*StoppableOnce) StopNow added in v0.1.0 

func (s *StoppableOnce) StopNow()

StopNow calls StopNow() on s.Stoppable, if StopNow() haven't been called yet.

type TaskContext 

type TaskContext struct {
	TaskInfo
	// contains filtered or unexported fields
}

The TaskContext exposes generic properties and functionality related to a task that is currently being executed.

This context is used to ensure that every component both engines and plugins that operates on a task have access to some common information about the task. This includes log drains, per-task credentials, generic task properties, and abortion notifications.

func (*TaskContext) Abort added in v0.0.2 

func (c *TaskContext) Abort()

Abort sets the status to aborted

func (*TaskContext) Authorizer added in v0.1.3 

func (c *TaskContext) Authorizer() client.Authorizer

Authorizer can sign requests with temporary credentials associated with the task.

Notice, when blindly forwarding requests task.scopes should be set as authorizedScopes, otherwise artifact upload and resolution will possible.

func (*TaskContext) Cancel added in v0.0.2 

func (c *TaskContext) Cancel()

Cancel sets the status to cancelled

func (*TaskContext) CreateErrorArtifact added in v0.0.11 

func (context *TaskContext) CreateErrorArtifact(artifact ErrorArtifact) error

CreateErrorArtifact is responsible for inserting error artifacts into the queue.

func (*TaskContext) CreateRedirectArtifact added in v0.0.11 

func (context *TaskContext) CreateRedirectArtifact(artifact RedirectArtifact) error

CreateRedirectArtifact is responsible for inserting redirect artifacts into the queue.

func (*TaskContext) Deadline added in v0.0.5 

func (c *TaskContext) Deadline() (deadline time.Time, ok bool)

Deadline returns empty time and false, this is implemented to satisfy context.Context.

func (*TaskContext) Done added in v0.0.5 

func (c *TaskContext) Done() <-chan struct{}

Done returns a channel that is closed when to TaskContext is aborted or canceled.

Implemented in compliance with context.Context.

func (*TaskContext) Err added in v0.0.5 

func (c *TaskContext) Err() error

Err returns context.Canceled, if task as canceled or aborted.

Implemented in compliance with context.Context.

func (*TaskContext) ExtractLog added in v0.0.2 

func (c *TaskContext) ExtractLog() (ioext.ReadSeekCloser, error)

ExtractLog returns an IO object to read the log.

func (*TaskContext) HasScopes added in v0.1.3 

func (c *TaskContext) HasScopes(scopeSets ...[]string) bool

HasScopes returns true, if task.scopes covers one of the scopeSets given

func (*TaskContext) IsAborted added in v0.0.2 

func (c *TaskContext) IsAborted() bool

IsAborted returns true if the current status is Aborted

func (*TaskContext) IsCancelled added in v0.0.2 

func (c *TaskContext) IsCancelled() bool

IsCancelled returns true if the current status is Cancelled

func (*TaskContext) Log 

func (c *TaskContext) Log(a ...interface{})

Log writes a log message from the worker

These log messages will be prefixed "[taskcluster]" so it's easy to see to that they are worker logs.

func (*TaskContext) LogDrain 

func (c *TaskContext) LogDrain() io.Writer

LogDrain returns a drain to which log message can be written.

Users should note that multiple writers are writing to this drain concurrently, and it is recommend that writers write in chunks of one line.

func (*TaskContext) LogError added in v0.0.2 

func (c *TaskContext) LogError(a ...interface{})

LogError writes a log error message from the worker

These log messages will be prefixed "[taskcluster:error]" so it's easy to see to that they are worker logs. These errors are also easy to grep from the logs in case of failure.

func (*TaskContext) NewLogReader 

func (c *TaskContext) NewLogReader() (io.ReadCloser, error)

NewLogReader returns a ReadCloser that reads the log from the start as the log is written.

Calls to Read() on the resulting ReadCloser are blocking. They will return when data is written or EOF is reached.

Consumers should ensure the ReadCloser is closed before discarding it.

func (*TaskContext) Queue added in v0.0.2 

func (c *TaskContext) Queue() client.Queue

Queue will return a client for the TaskCluster Queue. This client is useful for plugins that require interactions with the queue, such as creating artifacts.

func (*TaskContext) UploadS3Artifact added in v0.0.11 

func (context *TaskContext) UploadS3Artifact(artifact S3Artifact) error

UploadS3Artifact is responsible for creating new artifacts in the queue and then performing the upload to s3.

func (*TaskContext) Value added in v0.0.5 

func (c *TaskContext) Value(key interface{}) interface{}

Value returns nil, this is implemented to satisfy context.Context

type TaskContextController 

type TaskContextController struct {
	*TaskContext
}

TaskContextController exposes logic for controlling the TaskContext.

Spliting this out from TaskContext ensures that engines and plugins doesn't accidentally Dispose() the TaskContext.

func (*TaskContextController) CloseLog 

func (c *TaskContextController) CloseLog() error

CloseLog will close the log so no more messages can be written.

func (*TaskContextController) Dispose 

func (c *TaskContextController) Dispose() error

Dispose will clean-up all resources held by the TaskContext

func (*TaskContextController) SetCredentials added in v0.1.3 

func (c *TaskContextController) SetCredentials(clientID, accessToken, certificate string)

SetCredentials is used to provide the task-specific temporary credentials, and update these whenever they change.

func (*TaskContextController) SetQueueClient added in v0.0.2 

func (c *TaskContextController) SetQueueClient(client client.Queue)

SetQueueClient will set a client for the TaskCluster Queue. This client can then be used by others that have access to the task context and require interaction with the queue.

type TaskInfo 

type TaskInfo struct {
	TaskID   string
	RunID    int
	Created  time.Time
	Deadline time.Time
	Expires  time.Time
	Scopes   []string
	Task     interface{} // task definition in map[string]interface{} types..
}

The TaskInfo struct exposes generic properties from a task definition.

Note, do not be tempted to add task definition or status here in its entirety as it can encourage undesired behaviors. Instead only the data necessary should be exposed and nothing more. One such anti-pattern could be for a plugin to look at task.extra instead of adding data to task.payload.

type TaskStatus added in v0.0.2 

type TaskStatus string // TODO: (jonasfj) TaskContext shouldn't track status

TaskStatus represents the current status of the task. 

const (
	Aborted   TaskStatus = "Aborted" // TODO: (jonasfj) Don't distinguish between cancel/abort
	Cancelled TaskStatus = "Cancelled"
	Succeeded TaskStatus = "Succeeded"
	Failed    TaskStatus = "Failed"
	Errored   TaskStatus = "Errored"
	Claimed   TaskStatus = "Claimed"
	Reclaimed TaskStatus = "Reclaimed"
)

Enumerate task status to aid life-cycle decision making Use strings for benefit of simple logging/reporting

type TemporaryFile added in v0.0.2 

type TemporaryFile interface {
	Truncate(size int64) error
	io.ReadWriteSeeker
	io.Closer
	Path() string
}

TemporaryFile is a temporary file that will be removed when closed.

type TemporaryFolder added in v0.0.2 

type TemporaryFolder interface {
	TemporaryStorage
	Path() string
	Remove() error
}

TemporaryFolder is a temporary folder that is backed by the filesystem. User are nicely asked to stay with the folder they've been issued.

We don't really mock the file system interface as we need to integrate with other applications like docker, so we have to expose real file paths.

func NewTemporaryStorage added in v0.0.2 

func NewTemporaryStorage(path string) (TemporaryFolder, error)

NewTemporaryStorage return a TemporaryFolder rooted in the given path.

func NewTemporaryTestFolderOrPanic added in v0.1.0 

func NewTemporaryTestFolderOrPanic() TemporaryFolder

NewTemporaryTestFolderOrPanic creates a TemporaryFolder as in a subfolder of os.TempDir, or panics.

This intended to for use when writing tests using the following pattern: 

storage := runtime.NewTemporaryTestFolderOrPanic()
defer storage.Remove()

type TemporaryStorage added in v0.0.2 

type TemporaryStorage interface {
	NewFolder() (TemporaryFolder, error)
	NewFile() (TemporaryFile, error)
	NewFilePath() string
}

TemporaryStorage can create temporary folders and files.

Source Files

View all Source files

Directories

Path Synopsis
Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking.
 Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking.
Package caching provides an easy to make a cache on top of the gc package used to track idle resources in taskcluster-worker.
 Package caching provides an easy to make a cache on top of the gc package used to track idle resources in taskcluster-worker.
Package fetcher provides means for plugins and engines to fetch resources with generic references.
 Package fetcher provides means for plugins and engines to fetch resources with generic references.
Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.
 Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.
Package ioext contains interfaces and implementations for when the default io types are not sufficient.
 Package ioext contains interfaces and implementations for when the default io types are not sufficient.
Package mocks contains mock implementations of various interfaces useful for writing unit-tests.
 Package mocks contains mock implementations of various interfaces useful for writing unit-tests.
Package monitoring provides multiple implementations of runtime.Monitor.
 Package monitoring provides multiple implementations of runtime.Monitor.
Package util contains a few simple utilites that has no internal dependencies.
 Package util contains a few simple utilites that has no internal dependencies.
Package webhookserver provides implementations of the WebHookServer interface.
 Package webhookserver provides implementations of the WebHookServer interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.