drivers

README

This package is intended as a general purpose container abstraction library. With the same code, you can run on Docker, Rkt, etc.

Documentation

Overview

Package drivers is intended as a general purpose container abstraction library. It abstracts across the differences between different container runtimes (e.g. Docker, Rkt, etc.) and provides utlities and data types that are common across all runtimes.

Docker Driver

The docker driver runs functions as Docker containers.

Mock Driver

The mock driver pretends to run functions but doesn't actually run them. This is for testing only.

Index

• Constants
• func ParseImage(image string) (registry, repo, tag string)
• func Register(name string, driverFunc DriverFunc)
• type Config
• type ContainerTask
• type Cookie
• type Driver
  • func New(driverName string, config Config) (Driver, error)
• type DriverFunc
• type LoggerConfig
• type LoggerTag
• type RunResult
• type Stat
  • func Decimate(maxSamples int, stats []Stat) []Stat
• type Stats
  • func (s *Stats) Scan(value interface{}) error
  • func (s Stats) Value() (driver.Value, error)
• type WaitResult

Constants

const (
	// task statuses
	StatusRunning   = "running"
	StatusSuccess   = "success"
	StatusError     = "error"
	StatusTimeout   = "timeout"
	StatusKilled    = "killed"
	StatusCancelled = "cancelled"
)

TODO: ensure some type is applied to these statuses.

Functions

func ParseImage

func ParseImage(image string) (registry, repo, tag string)

func Register

func Register(name string, driverFunc DriverFunc)

Register adds a container driver by name to this process

Types

type Config

type Config struct {
	// TODO this should all be driver-specific config and not in the
	// driver package itself. fix if we ever one day try something else
	Docker               string `json:"docker"`
	DockerNetworks       string `json:"docker_networks"`
	DockerLoadFile       string `json:"docker_load_file"`
	ServerVersion        string `json:"server_version"`
	PreForkPoolSize      uint64 `json:"pre_fork_pool_size"`
	PreForkImage         string `json:"pre_fork_image"`
	PreForkCmd           string `json:"pre_fork_cmd"`
	PreForkUseOnce       uint64 `json:"pre_fork_use_once"`
	PreForkNetworks      string `json:"pre_fork_networks"`
	MaxTmpFsInodes       uint64 `json:"max_tmpfs_inodes"`
	EnableReadOnlyRootFs bool   `json:"enable_readonly_rootfs"`
	MaxRetries           uint64 `json:"max_retries"`
	ContainerLabelTag    string `json:"container_label_tag"`
	InstanceId           string `json:"instance_id"`
	ImageCleanMaxSize    uint64 `json:"image_clean_max_size"`
	ImageCleanExemptTags string `json:"image_clean_exempt_tags"`
}

type ContainerTask

type ContainerTask interface {
	// Command returns the command to run within the container.
	Command() string

	// EnvVars returns environment variable key-value pairs.
	EnvVars() map[string]string

	// Input feeds the container with data
	Input() io.Reader

	// The id to assign the container
	Id() string

	// Image returns the runtime specific image to run.
	Image() string

	// Timeout specifies the maximum time a task is allowed to run. Return 0 to let it run forever.
	Timeout() time.Duration

	// Driver will write output log from task execution to these writers. Must be
	// non-nil. Use io.Discard if log is irrelevant.
	Logger() (stdout, stderr io.Writer)

	// WriteStat writes a single Stat, implementation need not be thread safe.
	WriteStat(context.Context, Stat)

	// Volumes returns an array of 2-element tuples indicating storage volume mounts.
	// The first element is the path on the host, and the second element is the
	// path in the container.
	Volumes() [][2]string

	// Memory determines the max amount of RAM given to the container to use.
	// 0 is unlimited.
	Memory() uint64

	// CPUs in milli CPU units
	CPUs() uint64

	// Filesystem size limit for the container, in megabytes.
	FsSize() uint64

	// Tmpfs Filesystem size limit for the container, in megabytes.
	TmpFsSize() uint64

	// WorkDir returns the working directory to use for the task. Empty string
	// leaves it unset.
	WorkDir() string

	// Logger Config to use in driver
	LoggerConfig() LoggerConfig

	// Close is used to perform cleanup after task execution.
	// Close should be safe to call multiple times.
	Close()

	// Extensions are extra driver specific configuration options. They should be
	// more specific but it's easier to be lazy.
	Extensions() map[string]string

	// UDSAgentPath to use to configure the unix domain socket.
	// This is the mount point relative to the agent
	// abstractions have leaked so bad at this point it's a monsoon.
	UDSAgentPath() string

	// UDSDockerPath to use to configure the unix domain socket. the drivers
	// This is the mount point relative to the docker host.
	UDSDockerPath() string

	// UDSDockerDest is the destination mount point for uds path. it is the path
	// of the directory where the sock file resides inside of the container.
	UDSDockerDest() string
}

The ContainerTask interface guides container execution across a wide variety of container oriented runtimes.

type Cookie

type Cookie interface {
	// Close should clean up any resources the cookie was using, or was going to use.
	Close(ctx context.Context) error

	// Run should execute task on the implementation.
	// RunResult captures the result of task execution. This means if task
	// execution fails due to a problem in the task, Run() MUST return a valid
	// RunResult and nil as the error. The RunResult's Error() and Status()
	// should be used to indicate failure.
	// If the implementation itself suffers problems (lost of network, out of
	// disk etc.), a nil RunResult and an error message is preferred.
	//
	// Run() MUST monitor the context. task cancellation is indicated by
	// cancelling the context.
	Run(ctx context.Context) (WaitResult, error)

	// Freeze the container to pause running processes
	Freeze(ctx context.Context) error

	// Unfreeze a frozen container to unpause frozen processes
	Unfreeze(ctx context.Context) error

	// Authenticate image. Returns non-nil error if auth fails.
	AuthImage(ctx context.Context) error

	// Validate/Inspect image. Returns true if the image needs
	// to be pulled and non-nil error if validation/inspection fails.
	ValidateImage(ctx context.Context) (bool, error)

	// Pull the image. An image pull requires validation/inspection
	// again.
	PullImage(ctx context.Context) error

	// Create container which can be Run() later
	CreateContainer(ctx context.Context) error

	// Fetch driver specific container configuration. Use this to
	// access the container create options. If Driver.Prepare() is not
	// yet called with the cookie, then this can be used to modify container
	// create options.
	ContainerOptions() interface{}
}

A DriverCookie identifies a unique request to run a task.

Clients should always call Close() on a DriverCookie after they are done with it.

type Driver

type Driver interface {
	// Create a new cookie with defaults and/or settings from container task.
	// Callers should Close the cookie regardless of whether they prepare or run it.
	CreateCookie(ctx context.Context, task ContainerTask) (Cookie, error)

	// Obsoleted. No-Op
	PrepareCookie(ctx context.Context, cookie Cookie) error

	// close & shutdown the driver
	Close() error
}

func New

func New(driverName string, config Config) (Driver, error)

New Instantiates a driver by name

type DriverFunc

type DriverFunc func(config Config) (Driver, error)

type LoggerConfig

type LoggerConfig struct {
	// Log Sink URL
	URL string

	// Log Tag Pairs
	Tags []LoggerTag
}

Logger Configuration for container

type LoggerTag

type LoggerTag struct {
	Name  string
	Value string
}

Logger Tags for container

type RunResult

type RunResult interface {
	// Error is an actionable/checkable error from the container, nil if
	// Status() returns "success", otherwise non-nil
	Error() error

	// Status should return the current status of the task.
	// Only valid options are {"error", "success", "timeout", "killed", "cancelled"}.
	Status() string
}

RunResult indicates only the final state of the task.

type Stat

type Stat struct {
	Timestamp common.DateTime   `json:"timestamp"`
	Metrics   map[string]uint64 `json:"metrics"`
}

Stat is a bucket of stats from a driver at a point in time for a certain task.

func Decimate

func Decimate(maxSamples int, stats []Stat) []Stat

Decimate will down sample to a max number of points in a given sample by averaging samples together. i.e. max=240, if we have 240 samples, return them all, if we have 480 samples, every 2 samples average them (and time distance), and return 240 samples. This is relatively naive and if len(in) > max, <= max points will be returned, not necessarily max: length(out) = ceil(length(in)/max) -- feel free to fix this, setting a relatively high max will allow good enough granularity at higher lengths, i.e. for max of 1 hour tasks, sampling every 1s, decimate will return 15s samples if max=240. Large gaps in time between samples (a factor > (last-start)/max) will result in a shorter list being returned to account for lost samples. Decimate will modify the input list for efficiency, it is not copy safe. Input must be sorted by timestamp or this will fail gloriously.

type Stats

type Stats []Stat

Stats is a list of Stat, notably implements sql.Valuer

func (*Stats) Scan

func (s *Stats) Scan(value interface{}) error

implements sql.Scanner

func (Stats) Value

func (s Stats) Value() (driver.Value, error)

implements sql.Valuer, returning a string

type WaitResult

type WaitResult interface {
	// Wait may be called to await the result of a container's execution. If the
	// provided context is canceled and the container does not return first, the
	// resulting status will be 'canceled'. If the provided context times out
	// then the resulting status will be 'timeout'.
	Wait(context.Context) RunResult
}