backend

Documentation

Overview

Package backend provides the compute instance backends supported by Worker.

Other code will primarily interact with this package by creating a Provider implementation and creating Instances. An example using the "fake" provider (error handling omitted):

provider := backend.NewBackendProvider(
	"fake",
	config.ProviderConfigFromMap(map[string]string{
		"STARTUP_DURATION": "1s",
		"LOG_OUTPUT": "Hello, world!",
	}),
)

provider.Setup(ctx)

instance, _ := provider.Start(ctx, &backend.StartAttributes{
	Language: "go",
	OS: "linux",
})
defer instance.Stop(ctx)

instance.UploadScript(ctx, []byte("#!/bin/sh\necho 'Hello, world!'))
instance.RunScript(ctx, os.Stdout)

New providers should call Register in init() to register the alias it should be called with and the options it supports for the --help output.

Index

• Constants
• Variables
• func EachBackend(f func(*Backend))
• func Register(alias, humanReadableName string, providerHelp map[string]string, ...)
• type Backend
• type Instance
• type MetricsTransport
  • func (m *MetricsTransport) RoundTrip(req *http.Request) (*http.Response, error)
• type NullProgresser
  • func (np *NullProgresser) Progress(_ *ProgressEntry)
• type ProgressEntry
• type ProgressState
  • func (ps ProgressState) String() string
• type Progresser
• type Provider
  • func NewBackendProvider(alias string, cfg *config.ProviderConfig) (Provider, error)
• type RunResult
• type StartAttributes
  • func (sa *StartAttributes) SetDefaults(lang, dist, group, os, vmType string, vmConfig VmConfig)
• type TextProgresser
  • func NewTextProgresser(w io.Writer) *TextProgresser
  • func (tp *TextProgresser) Progress(entry *ProgressEntry)
• type VmConfig

Constants

const (

	// DockerMinSupportedAPIVersion of 1.24 means the client library in use here
	// can only support docker-engine 1.12 and above
	// (https://docs.docker.com/release-notes/docker-engine/#1120-2016-07-28).
	DockerMinSupportedAPIVersion = "1.24"
)

Variables

var (
	// ErrStaleVM is returned from one of the Instance methods if it detects
	// that the VM had already been used for a repository and was not reverted
	// afterwards.
	ErrStaleVM = fmt.Errorf("previous build artifacts found on stale vm")

	// ErrMissingEndpointConfig is returned if the provider config was missing
	// an 'ENDPOINT' configuration, but one is required.
	ErrMissingEndpointConfig = fmt.Errorf("expected config key endpoint")
)

var ErrDownloadTraceNotImplemented = errors.New("DownloadTrace not implemented")

Functions

func EachBackend

func EachBackend(f func(*Backend))

EachBackend calls a given function for each registered backend

func Register

func Register(alias, humanReadableName string, providerHelp map[string]string, providerFunc func(*config.ProviderConfig) (Provider, error))

Register adds a backend to the registry!

Types

type Backend

type Backend struct {
	Alias             string
	HumanReadableName string
	ProviderHelp      map[string]string
	ProviderFunc      func(*config.ProviderConfig) (Provider, error)
}

Backend wraps up an alias, backend provider help, and a factory func for a given backend provider wheee

type Instance

type Instance interface {
	// UploadScript uploads the given script to the instance. The script is
	// a bash script with a shebang (#!/bin/bash) line. Note that this
	// method should not be called multiple times.
	UploadScript(gocontext.Context, []byte) error

	// RunScript runs the build script that was uploaded with the
	// UploadScript method.
	RunScript(gocontext.Context, io.Writer) (*RunResult, error)

	// DownloadTrace attempts to download a job trace from the instance
	DownloadTrace(gocontext.Context) ([]byte, error)

	// Stop stops (and deletes) the instance
	Stop(gocontext.Context) error

	// ID is used when identifying the instance in logs and such
	ID() string

	// ImageName is the name of the image used to boot the instance
	ImageName() string

	// StartupDuration is the duration between "created" and "ready"
	StartupDuration() time.Duration

	// SupportsProgress allows for querying of progress support, yeah!
	SupportsProgress() bool

	// Check if this instance came from the warmer service
	Warmed() bool
}

An Instance is something that can run a build script.

type MetricsTransport

type MetricsTransport struct {
	Name      string
	Transport http.RoundTripper
}

func (*MetricsTransport) RoundTrip

func (m *MetricsTransport) RoundTrip(req *http.Request) (*http.Response, error)

type NullProgresser

type NullProgresser struct{}

func (*NullProgresser) Progress

func (np *NullProgresser) Progress(_ *ProgressEntry)

type ProgressEntry

type ProgressEntry struct {
	Message    string        `json:"message"`
	State      ProgressState `json:"state"`
	Interrupts bool          `json:"interrupts"`
	Continues  bool          `json:"continues"`
	Raw        bool          `json:"raw"`
}

type ProgressState

type ProgressState int

const (
	ProgressNeutral ProgressState = iota
	ProgressSuccess
	ProgressFailure
)

func (ProgressState) String

func (ps ProgressState) String() string

type Progresser

type Progresser interface {
	Progress(*ProgressEntry)
}

type Provider

type Provider interface {
	// Setup performs whatever is necessary in order to be ready to start
	// instances.
	Setup(gocontext.Context) error

	// Start starts an instance. It shouldn't return until the instance is
	// ready to call UploadScript on (this may, for example, mean that it
	// waits for SSH connections to be possible).
	Start(gocontext.Context, *StartAttributes) (Instance, error)

	// StartWithProgress starts an instance as with Start and also reports
	// progress via an io.Writer.
	StartWithProgress(gocontext.Context, *StartAttributes, Progresser) (Instance, error)

	// SupportsProgress allows for querying of progress support, yeah!
	SupportsProgress() bool
}

Provider represents some kind of instance provider. It can point to an external HTTP API, or some process locally, or something completely different.

func NewBackendProvider

func NewBackendProvider(alias string, cfg *config.ProviderConfig) (Provider, error)

NewBackendProvider looks up a backend by its alias and returns a provider via the factory func on the registered *Backend

type RunResult

type RunResult struct {
	// The exit code of the script. Only valid if Completed is true.
	ExitCode uint8

	// Whether the script finished running or not. Can be false if there was a
	// connection error in the middle of the script run.
	Completed bool
}

RunResult represents the result of running a script with Instance.RunScript.

type StartAttributes

type StartAttributes struct {
	Language  string `json:"language"`
	OsxImage  string `json:"osx_image"`
	Dist      string `json:"dist"`
	Group     string `json:"group"`
	OS        string `json:"os"`
	ImageName string `json:"image_name"`

	// The VMType isn't stored in the config directly, but in the top level of
	// the job payload, see the worker.JobPayload struct.
	VMType string `json:"-"`

	// The VMConfig isn't stored in the config directly, but in the top level of
	// the job payload, see the worker.JobPayload struct.
	VMConfig VmConfig `json:"-"`

	// Warmer isn't stored in the config directly, but in the top level of
	// the job payload, see the worker.JobPayload struct.
	Warmer bool `json:"-"`

	// HardTimeout isn't stored in the config directly, but is injected
	// from the processor
	HardTimeout time.Duration `json:"-"`

	// ProgressType isn't stored in the config directly, but is injected from
	// the processor
	ProgressType string `json:"-"`
}

StartAttributes contains some parts of the config which can be used to determine the type of instance to boot up (for example, what image to use)

func (*StartAttributes) SetDefaults

func (sa *StartAttributes) SetDefaults(lang, dist, group, os, vmType string, vmConfig VmConfig)

SetDefaults sets any missing required attributes to the default values provided

type TextProgresser

type TextProgresser struct {
	// contains filtered or unexported fields
}

func NewTextProgresser

func NewTextProgresser(w io.Writer) *TextProgresser

func (*TextProgresser) Progress

func (tp *TextProgresser) Progress(entry *ProgressEntry)

type VmConfig

type VmConfig struct {
	GpuCount int64  `json:"gpu_count"`
	GpuType  string `json:"gpu_type"`
	Zone     string `json:"zone"`
}