runtime

package
v0.0.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 27, 2017 License: MPL-2.0 Imports: 32 Imported by: 0

Documentation

Overview

Package runtime contains the generic functionality that an engine and plugins use.

Index

Constants

This section is empty.

Variables

View Source 
var ErrFatalInternalError = errors.New("Encountered a fatal internal error")

ErrFatalInternalError is used to signal that a fatal internal error has been logged and that the worker should gracefully terminate/reset.

Engines and plugins can return any unknown error in-order to trigger the same effect. As the worker will report, log and terminate/reset when it encounters an unknown error. This error is ONLY used when the error has already been reported and logged to both system log and task log.

This is only useful for plugins and engines that wishes to manually handle error reporting.

View Source 
var ErrLogClosed = errors.New("Log already closed")

ErrLogClosed raises when there is a double call to CloseLog.

View Source 
var ErrLogNotClosed = errors.New("Log is still open")

ErrLogNotClosed represents an invalid attempt to extract a log while it is still open.

View Source 
var ErrNonFatalInternalError = errors.New("Encountered a non-fatal internal error")

ErrNonFatalInternalError is used to indicate that the operation failed because of internal error that isn't expected to affect other tasks.

Worker need not worry about logging the error to system log or task log as the engine/plugin which returned this error already reported it, log it and/or deemed the error inconsequential.

Worker should, however, report the task as exception and resolve it with reason 'internal-error'. If the worker gets a lot of these non-fatal internal errors, it may employ a heuristic to decide if it has entered a bad state. For example, worker might reboot if it has seen more than 5 non-fatal internal errors within the span of 15min or 5 tasks.

Functions

func CreateLogger added in v0.0.2 

func CreateLogger(level string) (*logrus.Logger, error)

CreateLogger returns a new logger that can be passed around through the environment. Loggers can be created based on the one returned from this method by calling WithField or WithFields and specifying additional fields that the package would like.

func Gunzip added in v0.0.7 

func Gunzip(filename string) (string, error)

Gunzip gunzips a zipped file and returns its name

func NewTaskContext 

func NewTaskContext(tempLogFile string,
	task TaskInfo,
	server webhookserver.WebHookServer) (*TaskContext, *TaskContextController, error)

NewTaskContext creates a TaskContext and associated TaskContextController

func Untar added in v0.0.7 

func Untar(filename string) error

Untar unpacks a tar file

func Unzip added in v0.0.7 

func Unzip(filename string) error

Unzip unzips a zipped file

Types

type Environment 

type Environment struct {
	GarbageCollector gc.ResourceTracker
	TemporaryStorage
	webhookserver.WebHookServer
	Monitor
}

Environment is a collection of objects that makes up a runtime environment.

type ErrorArtifact added in v0.0.2 

type ErrorArtifact struct {
	Name    string
	Message string
	Reason  string
	Expires tcclient.Time
}

ErrorArtifact wraps all of the needed fields to upload an error artifact

type ExceptionReason added in v0.0.2 

type ExceptionReason string

An ExceptionReason specifies the reason a task reached an exception state. 

const (
	MalformedPayload ExceptionReason = "malformed-payload"
	WorkerShutdown   ExceptionReason = "worker-shutdown"
	InternalError    ExceptionReason = "internal-error"
)

Reasons why a task can reach an exception state. Implementors should be warned that additional entries may be added in the future.

type MalformedPayloadError added in v0.0.11 

type MalformedPayloadError struct {
	// contains filtered or unexported fields
}

The MalformedPayloadError error type is used to indicate that some operation failed because of malformed-payload.

For example a string expected to be path contained invalid characters, a required property was missing, or an integer was outside the permitted range.

func IsMalformedPayloadError added in v0.0.11 

func IsMalformedPayloadError(err error) (e MalformedPayloadError, ok bool)

IsMalformedPayloadError casts error to MalformedPayloadError.

This is mostly because it's hard to remember that error isn't supposed to be cast to *MalformedPayloadError.

func MergeMalformedPayload added in v0.0.11 

func MergeMalformedPayload(errors ...MalformedPayloadError) MalformedPayloadError

MergeMalformedPayload merges a list of MalformedPayloadError objects

func NewMalformedPayloadError added in v0.0.11 

func NewMalformedPayloadError(a ...interface{}) MalformedPayloadError

NewMalformedPayloadError creates a MalformedPayloadError object, please make sure to include a detailed description of the error, preferably using multiple lines and with examples.

These will be printed in the logs and end-users will rely on them to debug their tasks.

func (MalformedPayloadError) Error added in v0.0.11 

func (e MalformedPayloadError) Error() string

Error returns the error message and adheres to the Error interface

type Monitor added in v0.0.11 

type Monitor interface {
	// Measure values in statsum
	Measure(name string, value ...float64)
	// Increment counters in statsum
	Count(name string, value float64)
	// Measure time of fn in statsum
	Time(name string, fn func())

	// Report error/warning to sentry and write to log, returns incidentId which
	// can be included in task-logs, if relevant.
	ReportError(err error, message ...interface{}) string
	ReportWarning(err error, message ...interface{}) string

	// CapturePanic reports panics to log/sentry and returns incidentID, if any
	CapturePanic(fn func()) (incidentID string)

	// Write log messages to system log
	Debug(...interface{})
	Debugln(...interface{})
	Debugf(string, ...interface{})
	Print(...interface{})
	Println(...interface{})
	Printf(string, ...interface{})
	Info(...interface{})
	Infoln(...interface{})
	Infof(string, ...interface{})
	Warn(...interface{})
	Warnln(...interface{})
	Warnf(string, ...interface{})
	Error(...interface{})
	Errorln(...interface{})
	Errorf(string, ...interface{})
	Panic(...interface{})
	Panicln(...interface{})
	Panicf(string, ...interface{})

	// Create child monitor with given tags (tags don't apply to statsum)
	WithTags(tags map[string]string) Monitor
	WithTag(key, value string) Monitor
	// Create child monitor with given prefix (prefix applies to everything)
	WithPrefix(prefix string) Monitor
}

A Monitor is responsible for collecting logs, stats and error messages.

A monitor is a context aware object for monitoring. That is to say that a Monitor is used to record metrics, write logs and report errors. When doing so the Monitor object adds meta-data to the metrics, logs and errors. The meta-data added is context dependent tags and prefix. These help identify where a log message, metric or error originates from.

By encapsulating the context meta-data inside the Monitor object, an implementor gets a Monitor rarely needs to add tags or prefix. For example a monitor will always be prefixed by plugin name before being passed to a plugin, hence, it is easy trace any log message, metric or error report to the plugin that it was created in.

When passing a Monitor to a sub-component it often makes sense to add additional tags or prefix. This way a downloader function that takes a Monitor need not worry about being able to distinguish its metrics, logs and errors from that of its parent.

Prefixes should always be constants, such as engine, plugin, function or component names. Values that change such as taskId or runId should not be used as prefixes, such values is however great as tags.

All metrics reported for a given prefix + name will be aggregated. Hence, if taskId was used as prefix, the dimensionality of metrics would explode and the aggregation wouldn't be useful.

func NewLoggingMonitor added in v0.0.11 

func NewLoggingMonitor(logLevel string, tags map[string]string) Monitor

NewLoggingMonitor creates a monitor that just logs everything. This won't attempt to send anything to sentry or statsum.

func NewMonitor added in v0.0.11 

func NewMonitor(project string, auth client.Auth, logLevel string, tags map[string]string) Monitor

NewMonitor creates a new monitor

type RedirectArtifact added in v0.0.2 

type RedirectArtifact struct {
	Name     string
	Mimetype string
	URL      string
	Expires  tcclient.Time
}

RedirectArtifact wraps all of the needed fields to upload a redirect artifact

type S3Artifact added in v0.0.2 

type S3Artifact struct {
	Name              string
	Mimetype          string
	Expires           tcclient.Time
	Stream            ioext.ReadSeekCloser
	AdditionalHeaders map[string]string
}

S3Artifact wraps all of the needed fields to upload an s3 artifact

type ShutdownManager added in v0.0.2 

type ShutdownManager interface {
	WaitForShutdown() <-chan struct{}
}

ShutdownManager implements a method for listening for shutdown events. Consumers

func NewShutdownManager added in v0.0.2 

func NewShutdownManager(host string) ShutdownManager

NewShutdownManager will return a shutdown manager appropriate for the host that the worker is being run on.

Shutdown events are triggered different ways depending on where the worker is running. When running in AWS, then notifications are sent on their meta-data api, but running locally could cause the worker to represent to different kind of shutdown events.

type TaskContext 

type TaskContext struct {
	TaskInfo
	// contains filtered or unexported fields
}

The TaskContext exposes generic properties and functionality related to a task that is currently being executed.

This context is used to ensure that every component both engines and plugins that operates on a task have access to some common information about the task. This includes log drains, per-task credentials, generic task properties, and abortion notifications.

func (*TaskContext) Abort added in v0.0.2 

func (c *TaskContext) Abort()

Abort sets the status to aborted

func (*TaskContext) AttachWebHook added in v0.0.2 

func (c *TaskContext) AttachWebHook(handler http.Handler) string

AttachWebHook will take an http.Handler and expose it to the internet such that requests to any sub-resource of url returned will be forwarded to the handler.

Additionally, we promise that the URL contains a cryptographically random sequence of characters rendering it unpredictable. This can be used as a cheap form of access-control, and it is safe as task-specific web hooks are short-lived by nature.

Example use-cases:

  • livelog plugin
  • plugins for interactive shell/display/debugger, etc.
  • engines that send an email and await user confirmation ...

Implementors attaching a hook should take care to ensure that the handler is able to respond with a non-2xx response, if the data it is accessing isn't available anymore. All webhooks will be detached at the end of the task-cycle, but not until the very end.

func (*TaskContext) Cancel added in v0.0.2 

func (c *TaskContext) Cancel()

Cancel sets the status to cancelled

func (*TaskContext) CreateErrorArtifact added in v0.0.11 

func (context *TaskContext) CreateErrorArtifact(artifact ErrorArtifact) error

CreateErrorArtifact is responsible for inserting error artifacts into the queue.

func (*TaskContext) CreateRedirectArtifact added in v0.0.11 

func (context *TaskContext) CreateRedirectArtifact(artifact RedirectArtifact) error

CreateRedirectArtifact is responsible for inserting redirect artifacts into the queue.

func (*TaskContext) Deadline added in v0.0.5 

func (c *TaskContext) Deadline() (deadline time.Time, ok bool)

Deadline returns empty time and false, this is implemented to satisfy context.Context.

func (*TaskContext) Done added in v0.0.5 

func (c *TaskContext) Done() <-chan struct{}

Done returns a channel that is closed when to TaskContext is aborted or canceled.

Implemented in compliance with context.Context.

func (*TaskContext) Err added in v0.0.5 

func (c *TaskContext) Err() error

Err returns context.Canceled, if task as canceled or aborted.

Implemented in compliance with context.Context.

func (*TaskContext) ExtractLog added in v0.0.2 

func (c *TaskContext) ExtractLog() (ioext.ReadSeekCloser, error)

ExtractLog returns an IO object to read the log.

func (*TaskContext) IsAborted added in v0.0.2 

func (c *TaskContext) IsAborted() bool

IsAborted returns true if the current status is Aborted

func (*TaskContext) IsCancelled added in v0.0.2 

func (c *TaskContext) IsCancelled() bool

IsCancelled returns true if the current status is Cancelled

func (*TaskContext) Log 

func (c *TaskContext) Log(a ...interface{})

Log writes a log message from the worker

These log messages will be prefixed "taskcluster" so it's easy to see to that they are worker logs.

func (*TaskContext) LogDrain 

func (c *TaskContext) LogDrain() io.Writer

LogDrain returns a drain to which log message can be written.

Users should note that multiple writers are writing to this drain concurrently, and it is recommend that writers write in chunks of one line.

func (*TaskContext) LogError added in v0.0.2 

func (c *TaskContext) LogError(a ...interface{})

LogError writes a log error message from the worker

These log messages will be prefixed "[taskcluster:error]" so it's easy to see to that they are worker logs. These errors are also easy to grep from the logs in case of failure.

func (*TaskContext) NewLogReader 

func (c *TaskContext) NewLogReader() (io.ReadCloser, error)

NewLogReader returns a ReadCloser that reads the log from the start as the log is written.

Calls to Read() on the resulting ReadCloser are blocking. They will return when data is written or EOF is reached.

Consumers should ensure the ReadCloser is closed before discarding it.

func (*TaskContext) Queue added in v0.0.2 

func (c *TaskContext) Queue() client.Queue

Queue will return a client for the TaskCluster Queue. This client is useful for plugins that require interactions with the queue, such as creating artifacts.

func (*TaskContext) UploadS3Artifact added in v0.0.11 

func (context *TaskContext) UploadS3Artifact(artifact S3Artifact) error

UploadS3Artifact is responsible for creating new artifacts in the queue and then performing the upload to s3.

func (*TaskContext) Value added in v0.0.5 

func (c *TaskContext) Value(key interface{}) interface{}

Value returns nil, this is implemented to satisfy context.Context

type TaskContextController 

type TaskContextController struct {
	*TaskContext
}

TaskContextController exposes logic for controlling the TaskContext.

Spliting this out from TaskContext ensures that engines and plugins doesn't accidentally Dispose() the TaskContext.

func (*TaskContextController) CloseLog 

func (c *TaskContextController) CloseLog() error

CloseLog will close the log so no more messages can be written.

func (*TaskContextController) Dispose 

func (c *TaskContextController) Dispose() error

Dispose will clean-up all resources held by the TaskContext

func (*TaskContextController) SetQueueClient added in v0.0.2 

func (c *TaskContextController) SetQueueClient(client client.Queue)

SetQueueClient will set a client for the TaskCluster Queue. This client can then be used by others that have access to the task context and require interaction with the queue.

type TaskInfo 

type TaskInfo struct {
	TaskID   string
	RunID    int
	Created  tcclient.Time
	Deadline tcclient.Time
	Expires  tcclient.Time
}

The TaskInfo struct exposes generic properties from a task definition.

Note, do not be tempted to add task definition or status here in its entirety as it can encourage undesired behaviors. Instead only the data necessary should be exposed and nothing more. One such anti-pattern could be for a plugin to look at task.extra instead of adding data to task.payload.

type TaskStatus added in v0.0.2 

type TaskStatus string // TODO: (jonasfj) TaskContext shouldn't track status

TaskStatus represents the current status of the task. 

const (
	Aborted   TaskStatus = "Aborted" // TODO: (jonasfj) Don't distinguish between cancel/abort
	Cancelled TaskStatus = "Cancelled"
	Succeeded TaskStatus = "Succeeded"
	Failed    TaskStatus = "Failed"
	Errored   TaskStatus = "Errored"
	Claimed   TaskStatus = "Claimed"
	Reclaimed TaskStatus = "Reclaimed"
)

Enumerate task status to aid life-cycle decision making Use strings for benefit of simple logging/reporting

type TemporaryFile added in v0.0.2 

type TemporaryFile interface {
	io.ReadWriteSeeker
	io.Closer
	Path() string
}

TemporaryFile is a temporary file that will be removed when closed.

type TemporaryFolder added in v0.0.2 

type TemporaryFolder interface {
	TemporaryStorage
	Path() string
	Remove() error
}

TemporaryFolder is a temporary folder that is backed by the filesystem. User are nicely asked to stay with the folder they've been issued.

We don't really mock the file system interface as we need to integrate with other applications like docker, so we have to expose real file paths.

type TemporaryStorage added in v0.0.2 

type TemporaryStorage interface {
	NewFolder() (TemporaryFolder, error)
	NewFile() (TemporaryFile, error)
	NewFilePath() string
}

TemporaryStorage can create temporary folders and files.

func NewTemporaryStorage added in v0.0.2 

func NewTemporaryStorage(path string) (TemporaryStorage, error)

NewTemporaryStorage TemporaryStorage rooted in the given path.

Source Files

View all Source files

Directories

Path Synopsis
Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking.
 Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking.
Package fetcher provides means for plugins and engines to fetch resources with generic references.
 Package fetcher provides means for plugins and engines to fetch resources with generic references.
Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.
 Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.
Package ioext contains interfaces and implementations for when the default io types are not sufficient.
 Package ioext contains interfaces and implementations for when the default io types are not sufficient.
Package mocks contains mock implementations of various interfaces useful for writing unit-tests.
 Package mocks contains mock implementations of various interfaces useful for writing unit-tests.
Package util contains a few simple utilites that has no internal dependencies.
 Package util contains a few simple utilites that has no internal dependencies.
Package webhookserver provides implementations of the WebHookServer interface.
 Package webhookserver provides implementations of the WebHookServer interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.