plugins

package
v0.1.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 22, 2017 License: MPL-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package plugins defines interfaces to be implemented by feature plugins.

A plugin is an object that gets called for each step in the task execution flow. At each step the plugin will get access to engine and runtime object, such as the SandboxBuilder, the Sandbox, the SandboxContext, etc. A plugin is then supposed to implement a specific feature using these objects, this could be live logging, artifact uploading, attachment of proxies, mouting of caches, archival of caches, and many other things.

A plugin does not get any plugin specific task.payload space, instead the options required by a plugin must be part of the task.payload. Hence, all plugins must be available on all platforms. To facilitate this, plugins should not use platform specific APIs, instead they should rely on interface offered by the engine objects to do so, fail gracefully if an engine method is not supported in a given configuration and returns ErrFeatureNotSupported.

When a plugin encounters an unsupported feature that it needs, it may either return a MalformedPayloadError, or simply ignore the error and workaround it.

Plugin packages should provide a method: 

NewXXXPluginFactory(engine.Engine,*runtime.EngineContext) PluginFactory

and have it registered in pluginmanager.go

At high-level, PluginFactory stores the global state owned by the plugin, and Plugin stores the task-specific state owned by the plugin. A new Plugin instance will be created for each task.

In summary, plugins are not really "plugins", they are merely abstractions that allows us to implement features in complete isolation. Maybe they will become more flexible in the future, but there is no need to design for this at this point.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func PluginManagerConfigSchema added in v0.0.2 

func PluginManagerConfigSchema() schematypes.Object

PluginManagerConfigSchema returns configuration for PluginOptions.Config for NewPluginManager.

func Plugins added in v0.0.2 

func Plugins() map[string]PluginProvider

Plugins returns map from plugin name to PluginProviders.

func Register added in v0.0.2 

func Register(name string, provider PluginProvider)

Register will register a PluginProvider. This is meant to be called from init(), and will panic if name is already used by another plugin.

Types

type Plugin 

type Plugin interface {
	// Documentation returns a list of sections with end-user documentation.
	//
	// These sections will be combined with documentation sections from all
	// enabled plugins in-order to form end-user documentation.
	Documentation() []runtime.Section

	// PayloadSchema returns a schematypes.Object with the properties for
	// for the TaskPluginOptions.Payload property.
	//
	// Note: this will be merged with payload schemas from engine and other
	// plugins, thus, it cannot contain conflicting properties. Furthermore the
	// metadata will be discarded and additionalProperties will not be allowed.
	PayloadSchema() schematypes.Object

	// NewTaskPlugin method will be called once for each task. The TaskPlugin
	// instance returned will be called for each stage in the task execution.
	//
	// NewTaskPlugin will be called in parallel with NewSandboxBuilder(), making
	// it a good place to start long-running operations, you then have to take
	// care to clean-up in Dispose() if they are still running.
	// You should wait for your long-running operations to finish in
	//  BuildSandbox() or whatever hook you need them in.
	//
	// Plugins implementing logging should not return an error here, as it
	// naturally follows that such an error can't be logged if no logging plugin
	// is created.
	//
	// Implementors may return TaskPluginBase{}, if the plugin doesn't have any
	// hooks for the given tasks.
	//
	// Non-fatal errors: MalformedPayloadError
	NewTaskPlugin(options TaskPluginOptions) (TaskPlugin, error)

	// ReportIdle is called if the worker is idle prior to polling for tasks.
	// The durationSinceBusy is the time since the worker as last busy.
	//
	// This is a perfect time to initiate a graceful stop the worker by
	// calling runtime.Environment.Worker.StopGracefully(). A plugin can call
	// StopGracefully() at any time, causing the worker to stop polling.
	// This hook is mainly useful as a way to gracefully stop the worker without
	// entering a new billing cycle, which might happen if you gracefully stop
	// while the worker is processing one or more tasks.
	//
	// Warning: This hook is called as part of the worker claim loop, hence, it
	// strongly to execution time in this method
	ReportIdle(durationSinceBusy time.Duration)

	// ReportNonFatalError is called if the worker encountered a non-fatal error.
	//
	// Plugins can use this to implement a heuristic for shutting down after
	// certain number of non-fatal errors, or purging caches. This is mainly
	// intended to allow a configurable plugin to decide if the worker should
	// stop in response to a non-fatal error.
	//
	// The specific non-fatal error message is not available to the heuristic.
	// If a special heuristic is desired for a special non-fatal error, then this
	// should be handled in the plugin/engine where the error origins.
	ReportNonFatalError()

	// Dispose is called when the worker is stopping, a plugin should free all
	// resources and halt all background processes.
	//
	// This is important for testing, but also for relevant when running on
	// bare metal machines that aren't re-imaged between worker restarts.
	Dispose() error
}

Plugin is a plugin to the worker, for each task NewTaskPlugin is created. The Plugin instance is responsible for creating these objects and managing data shared between TaskPlugin instances.

All methods on this interface must be thread-safe.

type PluginBase 

type PluginBase struct{}

PluginBase is a base implementation of the Plugin interface, it just handles all methods and does nothing.

Implementors should embed this to ensure forward compatibility when we add new optional methods.

func (PluginBase) Dispose added in v0.1.0 

func (PluginBase) Dispose() error

Dispose does nothing

func (PluginBase) Documentation added in v0.1.0 

func (PluginBase) Documentation() []runtime.Section

Documentation returns no documentation.

func (PluginBase) NewTaskPlugin 

func (PluginBase) NewTaskPlugin(TaskPluginOptions) (TaskPlugin, error)

NewTaskPlugin returns TaskPluginBase{} which ignores all the stages.

func (PluginBase) PayloadSchema 

func (PluginBase) PayloadSchema() schematypes.Object

PayloadSchema returns a schema for an empty object for plugins that doesn't take any payload.

func (PluginBase) ReportIdle added in v0.1.0 

func (PluginBase) ReportIdle(time.Duration)

ReportIdle does nothing

func (PluginBase) ReportNonFatalError added in v0.1.0 

func (PluginBase) ReportNonFatalError()

ReportNonFatalError does nothing

type PluginManager added in v0.1.0 

type PluginManager struct {
	// contains filtered or unexported fields
}

A PluginManager combines a collection of plugins and implements an interface similar to Plugin. The interface is not exactly the same, notably, if there is an error, the PluginManager still returns a TaskPlugin wrapping plugins that didn't fail, such that artifacts and logs can be uploaded.

Furthermore, the PluginManager doesn't require that the task payload matches the PayloadSchema(), as such a requirement would preclude running logging plugins to upload logs for the task. Which is important as errors in the task log makes no sense, if the task log isn't uploaded.

func NewPluginManager added in v0.0.2 

func NewPluginManager(options PluginOptions) (*PluginManager, error)

NewPluginManager loads all plugins not disabled in configuration and returns a Plugin implementation that wraps all of the plugins.

This expects options.Config satisfying schema from PluginManagerConfigSchema().

Note. While the PluginManager implements the Plugin interface, it does have different semantics for NewTaskPlugin(). This is because we need to run other TaskPlugins (such as logging), even if one of the TaskPlugins fails to be created or there is a schema validation error.

func (*PluginManager) Dispose added in v0.1.0 

func (pm *PluginManager) Dispose() error

Dispose calls Dispose on all the managed plugins.

func (*PluginManager) Documentation added in v0.1.0 

func (pm *PluginManager) Documentation() []runtime.Section

Documentation will collect documentation from all managed plugins.

func (*PluginManager) NewTaskPlugin added in v0.1.0 

func (pm *PluginManager) NewTaskPlugin(options TaskPluginOptions) (TaskPlugin, error)

NewTaskPlugin constructs a TaskPlugin wrapping all the managed plugins whose PayloadSchema is satisfied by options.Payload.

This method always returns a TaskPlugin, even if there is errors. Because plugins that don't fail still want their Exception hook invoked.

func (*PluginManager) PayloadSchema added in v0.1.0 

func (pm *PluginManager) PayloadSchema() schematypes.Object

PayloadSchema returns the 'task.payload' schema expected by plugins.

func (*PluginManager) ReportIdle added in v0.1.0 

func (pm *PluginManager) ReportIdle(durationSinceBusy time.Duration)

ReportIdle call ReportIdle on all the managed plugins.

func (*PluginManager) ReportNonFatalError added in v0.1.0 

func (pm *PluginManager) ReportNonFatalError()

ReportNonFatalError calls ReportNonFatalError on all the managed plugins.

type PluginOptions added in v0.0.2 

type PluginOptions struct {
	Environment *runtime.Environment
	Engine      engines.Engine
	Monitor     runtime.Monitor
	Config      interface{}
}

PluginOptions is a wrapper for the arguments/options given when instantiating a Plugin using PluginProvider.

We wrap all arguments so that we can add additional properties without breaking source compatibility with older plugins.

type PluginProvider added in v0.0.2 

type PluginProvider interface {
	NewPlugin(options PluginOptions) (Plugin, error)

	// ConfigSchema returns schema for the PluginOptions.Config property.
	// May return nil, if no configuration should be given.
	ConfigSchema() schematypes.Schema
}

The PluginProvider interface must be implemented and registered by anyone implementing a Plugin.

If an implementor can determine that a plugin isn't available at compile-time it is preferred not to register the plugin.

type PluginProviderBase added in v0.0.2 

type PluginProviderBase struct{}

PluginProviderBase is a base struct that provides empty implementations of some methods for PluginProvider

Implementors of PluginProvider should embed this struct to ensure forward compatibility when we add new optional method to PluginProvider.

func (PluginProviderBase) ConfigSchema added in v0.0.2 

func (PluginProviderBase) ConfigSchema() schematypes.Schema

ConfigSchema returns a nil schema.

type TaskPlugin 

type TaskPlugin interface {
	// BuildSandbox is called once NewSandboxBuilder() has returned.
	//
	// This is the place to wait for downloads and other expensive operations to
	// finished, before mounting caches, proxies, etc. and returning.
	//
	// Non-fatal errors: MalformedPayloadError
	BuildSandbox(sandboxBuilder engines.SandboxBuilder) error

	// Started is called once the sandbox has started execution. This is a good
	// place to hook if you want to do interactive things.
	//
	// Non-fatal errors: MalformedPayloadError
	Started(sandbox engines.Sandbox) error

	// Stopped is called once the sandbox has terminated.
	//
	// This is a good place to upload artifacts, logs, check exit code, and start
	// to clean-up resources if such clean-up is expected to take a while.
	//
	// This will return false if the operation could not be completed successfully.
	// Such as artifact upload failure or files not existing.  If this returns false
	// it shall be assumed that the task has failed and should be reported as a failure.
	//
	// Non-fatal errors: MalformedPayloadError
	Stopped(result engines.ResultSet) (bool, error)

	// Finished is called once the sandbox has terminated and Stopped() have been
	// called.
	//
	// At this stage the task-specific log is closed, and attempts to log data
	// using the previous TaskContext will fail. That makes this a good place to
	// upload logs and any processing of the logs. In fact logging is the primary
	// motivation for this stage.
	//
	// As there is no logging in this method, it's not recommend to do anything
	// that may fail here.
	Finished(success bool) error

	// Exception is called once the task is resolved exception. This may happen
	// instead of calls to Prepare(), BuildSandbox(), Started(), Stopped(), or
	// Finished().
	//
	// This is a good place for best-effort to upload artifacts and logs that you
	// wish to persist. Naturally, log messages written at this stage will be
	// dropped and all error messages will be fatal.
	//
	// Implementors should be aware that additional reasons may be added in the
	// future. Therefore they must handle the default case, if switching on the
	// reason parameter.
	Exception(reason runtime.ExceptionReason) error

	// Dispose is called once everything is done and it's time for clean-up.
	//
	// This method will be invoked following Finished() or Exception(). It is then
	// the responsibility of the implementor to abort or wait for any long-running
	// processes and clean-up any resources held.
	Dispose() error
}

TaskPlugin holds the task-specific state for a plugin

Each method on this interface represents stage in the task execution and will be called when this stage is reached. The methods are allowed to take significant amounts of time, as they will run asynchronously.

These methods does not have to be thread-safe, we will never call the next method, before the previous method has returned.

Implementors of this interface should be sure to embed TaskPluginBase. This will do absolutely nothing, but provide empty implementations for any current and future methods that isn't implemented.

The methods are called in the order listed here with the exception of Exception() which may be called following any method, and Dispose() which will always be called as a final step allowing you to clean up.

type TaskPluginBase 

type TaskPluginBase struct{}

TaskPluginBase is a base implementation of the TaskPlugin interface, it just handles all methods and does nothing.

Implementors should embed this to ensure forward compatibility when we add new optional methods.

func (TaskPluginBase) BuildSandbox 

func (TaskPluginBase) BuildSandbox(engines.SandboxBuilder) error

BuildSandbox ignores the sandbox building stage.

func (TaskPluginBase) Dispose 

func (TaskPluginBase) Dispose() error

Dispose ignores the stage where resources are disposed.

func (TaskPluginBase) Exception 

func (TaskPluginBase) Exception(reason runtime.ExceptionReason) error

Exception ignores the stage where a task is resolved exception

func (TaskPluginBase) Finished 

func (TaskPluginBase) Finished(success bool) error

Finished ignores the stage where a task has been finished

func (TaskPluginBase) Started 

func (TaskPluginBase) Started(engines.Sandbox) error

Started ignores the stage where the sandbox has started

func (TaskPluginBase) Stopped 

func (TaskPluginBase) Stopped(engines.ResultSet) (bool, error)

Stopped ignores the stage where the sandbox has returned a ResultSet, and returns true saying the task was successful, as not to poison the water.

type TaskPluginOptions 

type TaskPluginOptions struct {
	TaskInfo    *runtime.TaskInfo
	TaskContext *runtime.TaskContext
	Payload     map[string]interface{}
	Monitor     runtime.Monitor
}

The TaskPluginOptions is a wrapper for the set of arguments given to NewTaskPlugin.

We wrap the arguments in a single argument to maintain source compatibility when introducing additional arguments.

Source Files

View all Source files

Directories

Path Synopsis
Package artifacts provides a taskcluster-worker plugin that uploads artifacts when sandbox execution has stopped.
 Package artifacts provides a taskcluster-worker plugin that uploads artifacts when sandbox execution has stopped.
Package env provides a taskcluster-worker plugin that injects environment variables into the task environments.
 Package env provides a taskcluster-worker plugin that injects environment variables into the task environments.
Package interactive implements the plugin that serves the interactive display and shell sessions over websockets.
 Package interactive implements the plugin that serves the interactive display and shell sessions over websockets.
displayclient
Package displayclient provides a golang implementation of websockify, transforming a websocket connection to an ioext.ReadWriteCloser object.
 Package displayclient provides a golang implementation of websockify, transforming a websocket connection to an ioext.ReadWriteCloser object.
displayconsts
pty
shellclient
Package shellclient provides a wrapper for demuxing a shell websocket and exposing the stdout/stderr streams as well as offering a way to provide the stdin stream.
 Package shellclient provides a wrapper for demuxing a shell websocket and exposing the stdout/stderr streams as well as offering a way to provide the stdin stream.
shellconsts
Package shellconsts contains constants shared between shell server and client which is split into different packages to reduce the binary size of potential commandline clients.
 Package shellconsts contains constants shared between shell server and client which is split into different packages to reduce the binary size of potential commandline clients.
Package livelog provides a taskcluster-worker plugin that makes the task log available as a live log during task execution and finally uploads it as a static log.
 Package livelog provides a taskcluster-worker plugin that makes the task log available as a live log during task execution and finally uploads it as a static log.
Package logprefix provides a taskcluster-worker plugin that prefixes all task logs with useful debug information such as taskId, workerType, as well as configurable constants.
 Package logprefix provides a taskcluster-worker plugin that prefixes all task logs with useful debug information such as taskId, workerType, as well as configurable constants.
Package maxruntime provides a plugin for taskcluster-worker which can enforce a maximum runtime upon tasks.
 Package maxruntime provides a plugin for taskcluster-worker which can enforce a maximum runtime upon tasks.
Package reboot provides a taskcluster-worker plugin that stops the worker after certain number of tasks or given amount of time.
 Package reboot provides a taskcluster-worker plugin that stops the worker after certain number of tasks or given amount of time.
Package stoponerror implements a very simple plugin that stops the worker gracefully if an non-fatal error is encountered.
 Package stoponerror implements a very simple plugin that stops the worker gracefully if an non-fatal error is encountered.
Package success implements a very simple plugin that looks that the ResultSet.Success() value to determine if the process from the sandbox exited successfully.
 Package success implements a very simple plugin that looks that the ResultSet.Success() value to determine if the process from the sandbox exited successfully.
Package tcproxy provides a taskcluster-worker plugin that exposes a proxy that signs requests with taskcluster credentials matching task.scopes.
 Package tcproxy provides a taskcluster-worker plugin that exposes a proxy that signs requests with taskcluster credentials matching task.scopes.
Package watchdog provides a taskcluster-worker plugin that pokes a watchdog whenever a task makes progress or the worker reports that it's idle.
 Package watchdog provides a taskcluster-worker plugin that pokes a watchdog whenever a task makes progress or the worker reports that it's idle.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.