hsup

package module
v0.0.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 15, 2015 License: BSD-2-Clause Imports: 46 Imported by: 0

README

hsup

Supervises processes that are configured in a Heroku-esque way.

hsup can poll the Heroku API directly to obtain releases, configuration, and execution information. hsup can also watch a local directory that injects similar information.

The execution is performed with a chosen "dyno driver":

  • The default dyno driver, simple, downloads and refreshes the environment only.
  • The docker dyno driver both obtains the environment and executable code and runs it interposed on the heroku/cedar:14 image.
  • The libcontainer driver is similar to the Docker driver, but runs containers in foreground, on top of Heroku official (read only) stack images. It needs to be executed as root (e.g.: sudo) and only works on Linux machines. See notes about running it in Docker (below) for nested (hsup-in-docker) support and execution on any host with Docker installed (e.g.: boot2docker).

Usage:

hsup COMMAND [options] [args...]

Where COMMAND is on one:

  • run: Run a command with an app's environment.
  • start: Start a process type as defined in an app's Procfile.

Example:

godep go install ./...
export DOCKER_HOST=unix:///var/run/docker.sock
export HEROKU_ACCESS_TOKEN=...

# start default process types inside Docker
hsup start -d docker start -a simple-brandur

# run command as subprocess
hsup run -d simple -a simple-brandur -- echo "hello"

Example using a directory:

godep go install ./...
export HSUP_CONTROL_DIR=/tmp/supctl
mkdir -p "$HSUP_CONTROL_DIR"
echo '{
    "Version": 1,
    "Env": {
        "NAME": "CONTENTS"
    },
    "Slug": "sample-slug.tgz",
    "Stack": "cedar-14",
    "Processes": [
        {
            "Args": ["./web-server", "arg"],
            "Quantity": 2,
            "Type": "web"
        },
        {
            "Args": ["./worker", "arg"],
            "Quantity": 2,
            "Type": "worker"
        }
    ]
}' > "$HSUP_CONTROL_DIR"/new
hsup run '/usr/bin/printenv'

# Note that after verifying the input, the file is moved to "loaded".
# Writing new "new" files is how updates can be issued.
ls "$HSUP_CONTROL_DIR"

Running the libcontainer driver within Docker

If you are using boot2docker, do the necessary preparation to expand the available disk space for hsup data:

$ make boot2docker-init

Containers (hsup/libcontainer) inside containers (docker). Inception!

$ docker build -t hsup .
$ docker run --privileged -it hsup
# will run /usr/bin/printenv, see docker/example.json for details

The docker container by default runs hsup start --oneshot. For a custom hsup command, use:

$ docker run --privileged -it hsup run bash
(dyno console) ~ $

Stack images will be downloaded for each new fresh container. It is a good idea to share a common stack image directory between all containers to avoid downloading them every time:

$ mkdir /tmp/stacks
$ docker run --privileged -v /tmp/stacks:/var/lib/hsup/stacks -it hsup
...
$ docker run --privileged -v /tmp/stacks:/var/lib/hsup/stacks -it hsup
(stack images will be cached)
...

A custom hsup control dir can be injected as a docker volume, in case custom json control files are required:

$ docker run --privileged -v /tmp/supctl:/etc/hsup -v /tmp/stacks:/var/lib/hsup/stacks -it hsup
(/tmp/supctl/new or /tmp/supctl/loaded will be used as the control file)

Automated functional tests

If you are using boot2docker (check the section above for details):

$ make boot2docker-init

To run several functional tests against a hsup binary:

$ godep go test ./ftest -driver docker -hsup <path-to-compiled-hsup-binary>

Different drivers (libcontainer, simple) can be specified with the driver flag, but note that the libcontainer driver requires root privileges:

$ sudo -E $(which godep) go test ./ftest -driver libcontainer -hsup <path-to-hsup-binary>

All tests can also be executed inside docker containers (see the hsup-inside-docker section above) with:

$ make ftest
runs libcontainer driver tests by default

$ make ftest driver=docker
specify a custom driver

$ make ftest-libcontainer
libcontainer driver tests...

$ make ftest-docker
docker driver tests...

$ make ftest-simple
simple driver tests...

Driver specific configuration

Some drivers accept custom configuration via ENV.

Docker
  • DOCKER_HOST
  • DOCKER_CERT_PATH
  • DOCKER_IMAGE_CACHE: when set, docker images are only built once per release.
Libcontainer
  • LIBCONTAINER_DYNO_SUBNET: a CIDR block to allocate dyno subnets (of size /30) from. It is 172.16.0.0/12 (RFC1918) by default when not set.
  • LIBCONTAINER_DYNO_UID_MIN and LIBCONTAINER_DYNO_UID_MAX: Linux UIDs to use for each dyno. It also defines the maximum number of allowed dynos, as each dyno gets a unique UID per box. To avoid reusing subnets (IPs), make sure that (maxUID - minUID) <= /30 subnets that LIBCONTAINER_DYNO_SUBNET can provide. 172.17.0.0/16 can provide 2 ** (30-16) = 16384 subnets of size /30. In this case, to avoid subnets being reused, make sure that (maxUID - minUID) <= 16384.

Documentation

Index

Constants

View Source
const (
	// currently available versions of Heroku stack images
	HerokuStacksManifestURL = "https://s3.amazonaws.com/heroku_stacks_production/manifest.yml"
)

Variables

View Source
var (
	// 172.16/12 block, starting at 172.16.0.28/30
	// By default allocate IPs from the RFC1918 (private address space),
	// which provides at most 2**18 = 262144 subnets of size /30.
	// Skip the first few IPs from RFC1918 to avoid clashes with
	// IPs used by AWS (eg.: the internal DNS server is 172.16.0.23 on EC2
	// classic).
	DefaultPrivateSubnet = net.IPNet{
		IP:   net.IPv4(172, 16, 0, 28).To4(),
		Mask: net.CIDRMask(12, 32),
	}
)
View Source
var ErrExecutorComplete = errors.New("Executor complete")
View Source
var (
	ErrInvalidIPMask = errors.New("mask is not a /30")
)
View Source
var ErrNoReleases = errors.New("No releases found")
View Source
var ErrNoSlugURL = errors.New("no slug specified")

Functions

func CurrentStackImagePath

func CurrentStackImagePath(stacksDir, name string) (string, error)

func StartControlAPI

func StartControlAPI(port int, processes <-chan *Processes) <-chan *Processes

Types

type APIFormation

type APIFormation struct {
	// contains filtered or unexported fields
}

func (*APIFormation) Args

func (f *APIFormation) Args() []string

func (*APIFormation) Quantity

func (f *APIFormation) Quantity() int

func (*APIFormation) Type

func (f *APIFormation) Type() string

type APIPoller

type APIPoller struct {
	Cl *heroku.Service
	Hs *Startup
	// contains filtered or unexported fields
}

func (*APIPoller) Notify

func (ap *APIPoller) Notify() <-chan *Processes

Listens for new releases by periodically polling the Heroku API. When a new release is detected it is sent to the returned channel.

type AbsPathDynoDriver

type AbsPathDynoDriver struct{}

func (*AbsPathDynoDriver) Build

func (dd *AbsPathDynoDriver) Build(release *Release) (err error)

func (*AbsPathDynoDriver) Start

func (dd *AbsPathDynoDriver) Start(ex *Executor) (err error)

func (*AbsPathDynoDriver) Stop

func (dd *AbsPathDynoDriver) Stop(ex *Executor) error

func (*AbsPathDynoDriver) Wait

func (dd *AbsPathDynoDriver) Wait(ex *Executor) (s *ExitStatus)

type Action added in v0.0.2

type Action int
const (
	Build Action = iota
	Start
	Run
)

type Allocator added in v0.0.7

type Allocator struct {
	// contains filtered or unexported fields
}

Allocator is responsible for allocating globally unique (per host) resources.

func NewAllocator added in v0.0.7

func NewAllocator(
	workDir string,
	privateSubnet net.IPNet,
	minUID, maxUID int,
) (*Allocator, error)

NewAllocator receives a CIDR block to allocate dyno subnets from, in the form baseIP/mask. All subnets will be >= baseIP, e.g.: 172.16.0.28/12 will cause subnets of size /30 to be allocated from 172.16/12, starting at 172.16.0.28/30.

To avoid reusing the same subnet for two different dynos (UIDs), make sure (maxUID - minUID) <= /30 subnets that the CIDR block can provide. E.g.: 172.17/16 can provide 2 ** (30-16) = 16384 /30 subnets, then to avoid subnets being reused, make sure that (maxUID - minUID) <= 16384.

func (*Allocator) FreeUID added in v0.0.7

func (a *Allocator) FreeUID(uid int) error

FreeUID returns the provided UID to the pool to be used by others

func (*Allocator) ReserveUID added in v0.0.7

func (a *Allocator) ReserveUID() (int, error)

ReserveUID optimistically locks uid numbers until one is successfully allocated. It relies on atomic filesystem operations to guarantee that multiple concurrent tasks will never allocate the same uid.

uid numbers allocated by this should be returned to the pool with FreeUID when they are not required anymore.

type AppSerializable

type AppSerializable struct {
	Version   int
	Name      string
	Env       map[string]string
	Slug      string
	Stack     string
	Processes []FormationSerializable

	// LogplexURL specifies where to forward the supervised
	// process Stdout and Stderr when non-empty.
	LogplexURL string `json:",omitempty"`
}

func (*AppSerializable) MustParseLogplexURL added in v0.0.5

func (as *AppSerializable) MustParseLogplexURL() *url.URL

Convenience function for parsing the stringy LogplexURL. This is helpful because gob encoding of url.URL values is not supported. It's presumed that the URL-conformance of LogplexURL has already been verified.

type ControlAPI

type ControlAPI struct {
	// contains filtered or unexported fields
}

func (*ControlAPI) ServeGET

func (c *ControlAPI) ServeGET(w http.ResponseWriter, r *http.Request)

func (*ControlAPI) ServeHTTP

func (c *ControlAPI) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*ControlAPI) ServePOST

func (c *ControlAPI) ServePOST(w http.ResponseWriter, r *http.Request)

func (*ControlAPI) Tee

func (c *ControlAPI) Tee(procs <-chan *Processes) <-chan *Processes

type DirPoller

type DirPoller struct {
	Dir string
	Hs  *Startup
	// contains filtered or unexported fields
}

func (*DirPoller) Notify

func (dp *DirPoller) Notify() <-chan *Processes

type Docker

type Docker struct {
	// contains filtered or unexported fields
}

func (*Docker) BuildSlugImage

func (d *Docker) BuildSlugImage(si *DockerStackImage, release *Release) (
	string, error)

func (*Docker) Connect

func (d *Docker) Connect() (err error)

func (*Docker) StackStat

func (d *Docker) StackStat(stack string) (*DockerStackImage, error)

type DockerDynoDriver

type DockerDynoDriver struct {
	// contains filtered or unexported fields
}

func (*DockerDynoDriver) Build

func (dd *DockerDynoDriver) Build(release *Release) error

func (*DockerDynoDriver) Start

func (dd *DockerDynoDriver) Start(ex *Executor) error

func (*DockerDynoDriver) Stop

func (dd *DockerDynoDriver) Stop(ex *Executor) error

func (*DockerDynoDriver) Wait

func (dd *DockerDynoDriver) Wait(ex *Executor) (s *ExitStatus)

type DockerStackImage

type DockerStackImage struct {
	// contains filtered or unexported fields
}

type DynoDriver

type DynoDriver interface {
	Build(*Release) error
	Start(*Executor) error
	Stop(*Executor) error
	Wait(*Executor) *ExitStatus
}

type DynoInput

type DynoInput int
const (
	Retire DynoInput = iota
	Restart
	Exited
	StayStarted
)

func (DynoInput) String

func (i DynoInput) String() string

type DynoState

type DynoState int
const (
	Stopped DynoState = iota
	Started
	Retiring
	Retired
)

func (DynoState) String

func (i DynoState) String() string

type Executor

type Executor struct {
	Args        []string
	DynoDriver  DynoDriver
	Release     *Release
	ProcessID   int
	ProcessType string
	Status      chan *ExitStatus
	Complete    chan struct{}
	LogplexURL  *url.URL
	Binds       map[string]string

	// FSM Fields
	OneShot  bool
	State    DynoState
	NewInput chan DynoInput
	// contains filtered or unexported fields
}

func (*Executor) Name

func (e *Executor) Name() string

func (*Executor) Tick

func (e *Executor) Tick() (err error)

func (*Executor) Trigger

func (e *Executor) Trigger(input DynoInput)

type ExitStatus

type ExitStatus struct {
	Code int
	Err  error
}

type Formation

type Formation interface {
	Args() []string
	Quantity() int
	Type() string
}

type FormationSerializable

type FormationSerializable struct {
	FArgs     []string `json:"Args"`
	FQuantity int      `json:"Quantity"`
	FType     string   `json:"Type"`
}

func (*FormationSerializable) Args

func (fs *FormationSerializable) Args() []string

func (*FormationSerializable) Quantity

func (fs *FormationSerializable) Quantity() int

func (*FormationSerializable) Type

func (fs *FormationSerializable) Type() string

type GobNotifier

type GobNotifier struct {
	Payload string
}

func (*GobNotifier) Notify

func (gn *GobNotifier) Notify() <-chan *Processes

type HerokuStackImage

type HerokuStackImage struct {
	Name    string
	Version string
	URL     string `yaml:"url"`
	Md5     string
	Primary bool
	// contains filtered or unexported fields
}

HerokuStackImage models stack images as they are distributed by Heroku: binary disk images, usually intended to be mounted on loopback devices. The common use is to mount those images read-only, so a single immutable image can be shared by multiple containers, saving disk space and memory due to Linux CoW page sharing.

Support for Heroku stack images is currently only enabled when building for Linux, because these images are currently only used by the libcontainer driver.

func HerokuStacksFromManifest

func HerokuStacksFromManifest(stacksDir string) ([]HerokuStackImage, error)

func (*HerokuStackImage) Dir

func (img *HerokuStackImage) Dir() string

func (*HerokuStackImage) Filename

func (img *HerokuStackImage) Filename() string

type LibContainerDynoDriver

type LibContainerDynoDriver struct {
	// contains filtered or unexported fields
}

func NewLibContainerDynoDriver

func NewLibContainerDynoDriver(workDir string) (*LibContainerDynoDriver, error)

func (*LibContainerDynoDriver) Build

func (dd *LibContainerDynoDriver) Build(release *Release) error

func (*LibContainerDynoDriver) Start

func (dd *LibContainerDynoDriver) Start(ex *Executor) error

func (*LibContainerDynoDriver) Stop

func (dd *LibContainerDynoDriver) Stop(ex *Executor) error

func (*LibContainerDynoDriver) Wait

func (dd *LibContainerDynoDriver) Wait(ex *Executor) (s *ExitStatus)

type LibContainerInitDriver

type LibContainerInitDriver struct{}

LibContainerInitDriver is intended to be used as init (PID=1) inside containers created by the libcontainer driver. It exists solely to be called by the libcontainer driver and should not be used directly.

func (*LibContainerInitDriver) Build

func (dd *LibContainerInitDriver) Build(*Release) error

func (*LibContainerInitDriver) Start

func (dd *LibContainerInitDriver) Start(ex *Executor) error

Start acts as PID=1 inside a container spawned by libcontainer, doing the required setup and re-exec'ing as the abspath driver

func (*LibContainerInitDriver) Stop

func (*LibContainerInitDriver) Wait

type Notifier

type Notifier interface {
	Notify() <-chan *Processes
}

type Processes

type Processes struct {
	Rel   *Release
	Forms []Formation

	Dd         DynoDriver
	OneShot    bool
	Executors  []*Executor
	LogplexURL *url.URL
}

type Release

type Release struct {
	// contains filtered or unexported fields
}

func (*Release) ConfigSlice

func (r *Release) ConfigSlice() []string

func (*Release) Name

func (r *Release) Name() string

func (*Release) Where

func (r *Release) Where() SlugWhere

type Routed added in v0.0.7

type Routed struct {
	network.Veth
	// contains filtered or unexported fields
}

Routed implements libcontainer's network.NetworkStrategy interface, offering containers only layer 3 connectivity to the outside world.

func (*Routed) Create added in v0.0.7

func (r *Routed) Create(
	config *network.Network, nspid int, state *network.NetworkState,
) error

Create sets up a veth pair, setting the config.Gateway address on the master (host) side. The veth pair forms a small subnet with a single host and gateway.

func (*Routed) Initialize added in v0.0.7

func (r *Routed) Initialize(
	net *network.Network, state *network.NetworkState,
) error

type SimpleDynoDriver

type SimpleDynoDriver struct {
}

func (*SimpleDynoDriver) Build

func (dd *SimpleDynoDriver) Build(release *Release) error

func (*SimpleDynoDriver) Start

func (dd *SimpleDynoDriver) Start(ex *Executor) error

func (*SimpleDynoDriver) Stop

func (dd *SimpleDynoDriver) Stop(ex *Executor) error

func (*SimpleDynoDriver) Wait

func (dd *SimpleDynoDriver) Wait(ex *Executor) (s *ExitStatus)

type SlugWhere

type SlugWhere int
const (
	Local SlugWhere = iota
	HTTP
)

type Startup added in v0.0.2

type Startup struct {
	// App contains a representation of a single release of an
	// application to run.
	App AppSerializable

	// OneShot is true when hsup terminates after the supervised
	// program exits.
	OneShot bool

	// StartNumber is the first allocated ProcessID.  e.g. "2" in
	// the case of "web.2".
	StartNumber int

	// Action enumerates the high level action of this hsup,
	// e.g. "run", "start", "build".
	Action Action

	// Driver specifies the DynoDriver used to execute a program
	// under hsup.  If used for sub-invocations, it must be
	// registered via "gob.Register".
	Driver DynoDriver

	// SkipBuild is set to true tos kip skip the build step of
	// running a program.  This is useful when hsup is being
	// executed in the context of an already-prepared image
	// containing a program.
	SkipBuild bool

	// Formation name for "Start" action.
	FormName string

	// ControlPort specifies the TCP port the hsup API listens on.
	// Set to nil when API support is disabled.
	ControlPort *int

	// For use with "run".
	Args []string

	// Binds enumerates paths bound from the host into a
	// container.
	Binds map[string]string
}

Startup is a serializable struct sufficient to perform sub-invocations of hsup.

func (*Startup) FromBase64Gob added in v0.0.2

func (hs *Startup) FromBase64Gob(payload string)

func (*Startup) Procs added in v0.0.2

func (hs *Startup) Procs() *Processes

func (*Startup) ToBase64Gob added in v0.0.2

func (hs *Startup) ToBase64Gob() string

type StatusResponse

type StatusResponse struct {
	Processes map[string]string
}

type StopRequest

type StopRequest struct {
	Processes []string
}

type StopResponse

type StopResponse struct {
	StoppedProcesses []string
}

Directories

Path Synopsis
Godeps
_workspace/src/bitbucket.org/kardianos/osext
Extensions to the standard "os" package.
Extensions to the standard "os" package.
_workspace/src/code.google.com/p/go-uuid/uuid
The uuid package generates and inspects UUIDs.
The uuid package generates and inspects UUIDs.
_workspace/src/github.com/bmizerany/aws4
Package aws4 signs HTTP requests as prescribed in http://docs.amazonwebservices.com/general/latest/gr/signature-version-4.html
Package aws4 signs HTTP requests as prescribed in http://docs.amazonwebservices.com/general/latest/gr/signature-version-4.html
_workspace/src/github.com/bmizerany/aws4/dydb
This is an experimental library for use with DynamoDB.
This is an experimental library for use with DynamoDB.
_workspace/src/github.com/coreos/go-systemd/dbus
Integration with the systemd D-Bus API.
Integration with the systemd D-Bus API.
_workspace/src/github.com/cyberdelia/heroku-go/v3
Generated service client for heroku API.
Generated service client for heroku API.
_workspace/src/github.com/docker/docker/pkg/pools
Package pools provides a collection of pools which provide various data types with buffers.
Package pools provides a collection of pools which provide various data types with buffers.
_workspace/src/github.com/docker/libcontainer
Temporary API endpoint for libcontainer while the full API is finalized (api.go).
Temporary API endpoint for libcontainer while the full API is finalized (api.go).
_workspace/src/github.com/docker/libcontainer/integration
integration is used for integration testing of libcontainer
integration is used for integration testing of libcontainer
_workspace/src/github.com/docker/libcontainer/netlink
Packet netlink provide access to low level Netlink sockets and messages.
Packet netlink provide access to low level Netlink sockets and messages.
_workspace/src/github.com/fsouza/go-dockerclient
Package docker provides a client for the Docker remote API.
Package docker provides a client for the Docker remote API.
_workspace/src/github.com/fsouza/go-dockerclient/testing
Package testing provides a fake implementation of the Docker API, useful for testing purpose.
Package testing provides a fake implementation of the Docker API, useful for testing purpose.
_workspace/src/github.com/godbus/dbus
Package dbus implements bindings to the D-Bus message bus system.
Package dbus implements bindings to the D-Bus message bus system.
_workspace/src/github.com/godbus/dbus/introspect
Package introspect provides some utilities for dealing with the DBus introspection format.
Package introspect provides some utilities for dealing with the DBus introspection format.
_workspace/src/github.com/godbus/dbus/prop
Package prop provides the Properties struct which can be used to implement org.freedesktop.DBus.Properties.
Package prop provides the Properties struct which can be used to implement org.freedesktop.DBus.Properties.
_workspace/src/github.com/nu7hatch/gouuid
This package provides immutable UUID structs and the functions NewV3, NewV4, NewV5 and Parse() for generating versions 3, 4 and 5 UUIDs as specified in RFC 4122.
This package provides immutable UUID structs and the functions NewV3, NewV4, NewV5 and Parse() for generating versions 3, 4 and 5 UUIDs as specified in RFC 4122.
_workspace/src/github.com/ogier/pflag
pflag is a drop-in replacement for Go's flag package, implementing POSIX/GNU-style --flags.
pflag is a drop-in replacement for Go's flag package, implementing POSIX/GNU-style --flags.
_workspace/src/github.com/rcrowley/go-metrics
Go port of Coda Hale's Metrics library <https://github.com/rcrowley/go-metrics> Coda Hale's original work: <https://github.com/codahale/metrics>
Go port of Coda Hale's Metrics library <https://github.com/rcrowley/go-metrics> Coda Hale's original work: <https://github.com/codahale/metrics>
Metrics output to StatHat.
_workspace/src/github.com/syndtr/gocapability/capability
Package capability provides utilities for manipulating POSIX capabilities.
Package capability provides utilities for manipulating POSIX capabilities.
_workspace/src/gopkg.in/yaml.v2
Package yaml implements YAML support for the Go language.
Package yaml implements YAML support for the Go language.
cmd
Package diag implements a fixed-size ring buffer to store diagnostic text in.
Package diag implements a fixed-size ring buffer to store diagnostic text in.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL