README
¶
TaskCluster Worker
[](http://travis-ci.org/taskcluster/taskcluster-worker)
[](https://godoc.org/github.com/taskcluster/taskcluster-worker)
[](https://coveralls.io/github/taskcluster/taskcluster-worker?branch=master)
[](http://mozilla.org/MPL/2.0)
A worker for TaskCluster, written in go.
This is our next generation worker, that has a pluggable architecture for adding support for new engines (think Docker™ engine, Windows™ native engine, OS X™ native engine, KVM™/Xen™ engine) and adding engine-independent plugins (think livelogs, caches/volumes, auth proxies, interactive ssh/vnc).
Architecture
See https://docs.taskcluster.net/manual/execution/workers/taskcluster-worker
Installing From Binary
See https://github.com/taskcluster/taskcluster-worker/releases
Installing From Source
- Install go 1.7 or higher
go get -u -t -d github.com/taskcluster-worker/...cd "${GOPATH}/src/github.com/taskcluster-worker"make rebuild
Testing
make rebuild
Releasing
Simply create a tag, and push to github.
git tag v1.0.3
git push --tags
Freezing Dependencies
You need govendor to manage vendor dependencies.
go get -u https://github.com/kardianos/govendor # install govendor tool
govendor sync
Adding Dependencies
go get <package>
govendor add +ext <package>
git add vendor/vendor.json
git commit -m 'My new package.'
Updating Dependencies
go get -u -t ./... # update versions
govendor update
Contributing
We welcome Pull Requests and Issues!
Find us in #taskcluster-worker on irc.mozilla.org
Documentation
¶
Overview ¶
Package main hosts the main function for taskcluter-worker.
The code is structured in 4 kinds of extension registries. The commands, config, engines and plugins packages each define interfaces and registries where implementations of these interfaces can be registered.
To add a new command to taskcluster-worker you must create new sub-package of commands/ which implements and registers commands.CommandProvider with the commands.Register(name, provider) method. The same pattern is followed for implementation of config transformers, engines, and plugins.
All the sub-packages are then imported here, which ensure that they'll all be included in the respective extension registries. Exceptions to this pattern is the runtime and worker packages. The runtime package and its sub-packages implements generic abstractions and utilities to be used by all other packages. The worker package implements task execution flow to be used by commands.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package commands exposes a run method for main() to call
|
Package commands exposes a run method for main() to call |
|
help
Package help provides the help command.
|
Package help provides the help command. |
|
qemu-guest-tools
Package qemuguesttools implements the command that runs inside a QEMU VM.
|
Package qemuguesttools implements the command that runs inside a QEMU VM. |
|
shell
Package shell provides a CommandProvider that implements a CLI tool for opening to a interactive shell to an interactive taskcluster-worker task in your terminal.
|
Package shell provides a CommandProvider that implements a CLI tool for opening to a interactive shell to an interactive taskcluster-worker task in your terminal. |
|
Package config provides configuration loading logic.
|
Package config provides configuration loading logic. |
|
abs
Package configabs implements a TransformationProvider that replaces objects on the form: {$abs: "path"} with the value of current working folder + path.
|
Package configabs implements a TransformationProvider that replaces objects on the form: {$abs: "path"} with the value of current working folder + path. |
|
configtest
Package configtest provides structs and logic for declarative configuration tests.
|
Package configtest provides structs and logic for declarative configuration tests. |
|
env
Package configenv implements a TransformationProvider that replaces objects on the form: {$env: "VAR"} with the value of the environment variable VAR.
|
Package configenv implements a TransformationProvider that replaces objects on the form: {$env: "VAR"} with the value of the environment variable VAR. |
|
packet
Package configpacket implements a TransformationProvider that replaces objects on the form: {$packet: "VARIABLE"} with a value loaded from https://metadata.packet.net/metadata, following VARIABLE values are supported: - instance-id - hostname - facility - instance-type - public-ipv4 - public-ipv6 If configuration property 'packetMetaDataUrl' this will be used instead of 'https://metadata.packet.net/metadata'.
|
Package configpacket implements a TransformationProvider that replaces objects on the form: {$packet: "VARIABLE"} with a value loaded from https://metadata.packet.net/metadata, following VARIABLE values are supported: - instance-id - hostname - facility - instance-type - public-ipv4 - public-ipv6 If configuration property 'packetMetaDataUrl' this will be used instead of 'https://metadata.packet.net/metadata'. |
|
secrets
Package configsecrets implements a TransformationProvider that replaces objects on the form: {$secret: "NAME", key: "KEY"} with the value of the key "KEY" taken from the secret NAME loaded from taskcluster-secrets.
|
Package configsecrets implements a TransformationProvider that replaces objects on the form: {$secret: "NAME", key: "KEY"} with the value of the key "KEY" taken from the secret NAME loaded from taskcluster-secrets. |
|
Package engines specifies the interfaces that each engine must implement.
|
Package engines specifies the interfaces that each engine must implement. |
|
enginetest
Package enginetest provides utilities for testing generic engine implementations.
|
Package enginetest provides utilities for testing generic engine implementations. |
|
mock
Package mockengine implements a MockEngine that doesn't really do anything, but allows us to test plugins without having to run a real engine.
|
Package mockengine implements a MockEngine that doesn't really do anything, but allows us to test plugins without having to run a real engine. |
|
mock/mocknet
Package mocknet implements a net.Listener interface that can reached with mocknet.Dial() and establishes connections using net.Pipe() This is useful for testing things that needs net.Listener and net.Conn instances without creating a TCP listener on localhost.
|
Package mocknet implements a net.Listener interface that can reached with mocknet.Dial() and establishes connections using net.Pipe() This is useful for testing things that needs net.Listener and net.Conn instances without creating a TCP listener on localhost. |
|
native
Package nativeengine provides an engine with minimal sandboxing relying on per-task user accounts, temporary folders and process isolation.
|
Package nativeengine provides an engine with minimal sandboxing relying on per-task user accounts, temporary folders and process isolation. |
|
native/system
Package system implements cross-platform abstractions for user-management access-control and sub-process execution geared at executing sub-process with best-effort sandboxing.
|
Package system implements cross-platform abstractions for user-management access-control and sub-process execution geared at executing sub-process with best-effort sandboxing. |
|
osxnative
Package osxnative implements the Mac OSX engine
|
Package osxnative implements the Mac OSX engine |
|
qemu
Package qemuengine implements a QEMU based engine for taskcluster-worker.
|
Package qemuengine implements a QEMU based engine for taskcluster-worker. |
|
qemu/image
Package image exposes methods and abstractions for extracting and managing virtual machine images.
|
Package image exposes methods and abstractions for extracting and managing virtual machine images. |
|
qemu/metaservice
Package metaservice implements the meta-data service that the guests use to talk to the host.
|
Package metaservice implements the meta-data service that the guests use to talk to the host. |
|
qemu/network
Package network contains scripts and abstractions for setting up TAP-device based networks for a set of QEMU virtual machines.
|
Package network contains scripts and abstractions for setting up TAP-device based networks for a set of QEMU virtual machines. |
|
qemu/vm
Package vm provides virtual machine abstractions for QEMU.
|
Package vm provides virtual machine abstractions for QEMU. |
|
script
Package scriptengine provides an engine that can be configured with a script and a JSON schema, such that the worker executes declarative tasks.
|
Package scriptengine provides an engine that can be configured with a script and a JSON schema, such that the worker executes declarative tasks. |
|
Package plugins defines interfaces to be implemented by feature plugins.
|
Package plugins defines interfaces to be implemented by feature plugins. |
|
artifacts
Package artifacts is responsible for uploading artifacts after builds
|
Package artifacts is responsible for uploading artifacts after builds |
|
interactive
Package interactive implements the plugin that serves the interactive display and shell sessions over websockets.
|
Package interactive implements the plugin that serves the interactive display and shell sessions over websockets. |
|
interactive/displayclient
Package displayclient provides a golang implementation of websockify, transforming a websocket connection to an ioext.ReadWriteCloser object.
|
Package displayclient provides a golang implementation of websockify, transforming a websocket connection to an ioext.ReadWriteCloser object. |
|
interactive/shellclient
Package shellclient provides a wrapper for demuxing a shell websocket and exposing the stdout/stderr streams as well as offering a way to provide the stdin stream.
|
Package shellclient provides a wrapper for demuxing a shell websocket and exposing the stdout/stderr streams as well as offering a way to provide the stdin stream. |
|
interactive/shellconsts
Package shellconsts contains constants shared between shell server and client which is split into different packages to reduce the binary size of potential commandline clients.
|
Package shellconsts contains constants shared between shell server and client which is split into different packages to reduce the binary size of potential commandline clients. |
|
livelog
Package livelog implements a webhook handler for serving up livelogs of a task sandbox.
|
Package livelog implements a webhook handler for serving up livelogs of a task sandbox. |
|
success
Package success implements a very simple plugin that looks that the ResultSet.Success() value to determine if the process from the sandbox exited successfully.
|
Package success implements a very simple plugin that looks that the ResultSet.Success() value to determine if the process from the sandbox exited successfully. |
|
Package runtime contains the generic functionality that an engine and plugins use.
|
Package runtime contains the generic functionality that an engine and plugins use. |
|
atomics
Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking.
|
Package atomics provides types that can be concurrently accessed and modified, without caller code needing to implement locking. |
|
fetcher
Package fetcher provides means for plugins and engines to fetch resources with generic references.
|
Package fetcher provides means for plugins and engines to fetch resources with generic references. |
|
gc
Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources.
|
Package gc contains the GarbageCollector which allows cacheable resources to register themselves for disposal when we run low on resources. |
|
ioext
Package ioext contains interfaces and implementations for when the default io types are not sufficient.
|
Package ioext contains interfaces and implementations for when the default io types are not sufficient. |
|
util
Package util contains a few simple utilites that has no internal dependencies.
|
Package util contains a few simple utilites that has no internal dependencies. |
|
webhookserver
Package webhookserver provides implementations of the WebHookServer interface.
|
Package webhookserver provides implementations of the WebHookServer interface. |
|
Package worker is responsible for managing the claiming, executing, and resolution of tasks.
|
Package worker is responsible for managing the claiming, executing, and resolution of tasks. |
Click to show internal directories.
Click to hide internal directories.