dexec

README

dexec

dexec is a small Go library allowing you to run processes inside Docker containers as if you are running them locally using os/exec package. Read documentation at godoc.org or see examples.

Using dexec, you can do stream processing, off-load computationally expensive parts of your program to a remote fleet of Docker engines. Its interface is strikingly similar to os/exec.

Examples

Check out the following examples:

• Hello, world inside container →
• Connect to container’s STDIN/STDOUT →
• Stream processing with pipes →
• Check exit code of a remote process →
• Audio extraction from YouTube videos →
• Parallel computation on Swarm →

A Short Example

It takes only a 4-line code change to convert a piece of code using os/exec to use dexec to start running your stuff inside containers.

Here is a minimal Go program that runs echo in a container:

package main

import (
	"log"

	"github.com/ahmetalpbalkan/dexec"
	"github.com/fsouza/go-dockerclient"
)

func main(){
	cl, _ := docker.NewClient("unix:///var/run/docker.sock")
	d := dexec.Docker{cl}

	m, _ := dexec.ByCreatingContainer(docker.CreateContainerOptions{
	Config: &docker.Config{Image: "busybox"}})

	cmd := d.Command(m, "echo", `I am running inside a container!`)
	b, err := cmd.Output()
	if err != nil { log.Fatal(err) }
	log.Printf("%s", b)
}

Output: I am running inside a container!

Use Cases

This library is intended for providing an execution model that looks and feels like os/exec package.

You might want to use this library when:

• You want to execute a process, but run it in a container with extra security and finer control over resource usage with Docker –and change your code minimally.

• You want to execute a piece of work on a remote machine (or even better, a pool of machines or a cluster) through Docker. Especially useful to distribute computationally expensive workloads.

For such cases, this library abstracts out the details of executing the process in a container and gives you a cleaner interface you are already familiar with.

Check out more examples →

Read more →

Documentation

Overview

Package dexec provides an interface similar to os/exec to run external commands inside containers.

Please read documentation carefully about semantic differences between os/exec and dexec.

Use Case

This utility is intended to provide an execution environment without changing the existing code a lot to execute a command on a remote machine (or a pool of machines) running Docker engine or locally to limit resource usage and have extra security.

Dependencies

The package needs the following dependencies to work:

go get github.com/fsouza/go-dockerclient

Known issues

- You may receive empty stdout/stderr from commands if the executed command does not end with a trailing new line or has a different flushing behavior.

- StdinPipe/StdoutPipe/StderrPipe should be used with goroutines (one that executes Cmd.Wait() and another one that writes to/reads from the pipe. Otherwise, the code may hang 10% of the time due to some timing issue.

Index

• type Cmd
  • func (c *Cmd) CombinedOutput() ([]byte, error)
  • func (c *Cmd) Output() ([]byte, error)
  • func (c *Cmd) Run() error
  • func (c *Cmd) Start() error
  • func (c *Cmd) StderrPipe() (io.ReadCloser, error)
  • func (c *Cmd) StdinPipe() (io.WriteCloser, error)
  • func (c *Cmd) StdoutPipe() (io.ReadCloser, error)
  • func (c *Cmd) Wait() error
• type Docker
  • func (d Docker) Command(method Execution, name string, arg ...string) *Cmd
• type Execution
  • func ByCreatingContainer(opts docker.CreateContainerOptions) (Execution, error)
• type ExitError
  • func (e *ExitError) Error() string

Examples

• Cmd.Output

Types

type Cmd

type Cmd struct {
	// Method provides the execution strategy for the context of the Cmd.
	// An instance of Method should not be reused between Cmds.
	Method Execution

	// Path is the path or name of the command in the container.
	Path string

	// Arguments to the command in the container, excluding the command
	// name as the first argument.
	Args []string

	// Env is environment variables to the command. If Env is nil, Run will use
	// Env specified on Method or pre-built container image.
	Env []string

	// Dir specifies the working directory of the command. If Dir is the empty
	// string, Run uses Dir specified on Method or pre-built container image.
	Dir string

	// Stdin specifies the process's standard input.
	// If Stdin is nil, the process reads from the null device (os.DevNull).
	//
	// Run will not close the underlying handle if the Reader is an *os.File
	// differently than os/exec.
	Stdin io.Reader

	// Stdout and Stderr specify the process's standard output and error.
	// If either is nil, they will be redirected to the null device (os.DevNull).
	//
	// Run will not close the underlying handles if they are *os.File differently
	// than os/exec.
	Stdout io.Writer
	Stderr io.Writer
	// contains filtered or unexported fields
}

Cmd represents an external command being prepared or run.

A Cmd cannot be reused after calling its Run, Output or CombinedOutput methods.

func (*Cmd) CombinedOutput

func (c *Cmd) CombinedOutput() ([]byte, error)

CombinedOutput runs the command and returns its combined standard output and standard error.

Docker API does not have strong guarantees over ordering of messages. For instance:

>&1 echo out; >&2 echo err

may result in "out\nerr\n" as well as "err\nout\n" from this method.

func (*Cmd) Output

func (c *Cmd) Output() ([]byte, error)

Output runs the command and returns its standard output.

If the container exits with a non-zero exit code, the error is of type *ExitError. Other error types may be returned for I/O problems and such.

If c.Stderr was nil, Output populates ExitError.Stderr.

Example

cl, _ := docker.NewClient("unix:///var/run/docker.sock")
d := dexec.Docker{cl}

m, _ := dexec.ByCreatingContainer(docker.CreateContainerOptions{
	Config: &docker.Config{Image: "busybox"}})

cmd := d.Command(m, "echo", `I am running inside a container!`)
b, err := cmd.Output()
if err != nil {
	log.Fatal(err)
}
fmt.Printf("%s", b)

Output:

I am running inside a container!

func (*Cmd) Run

func (c *Cmd) Run() error

Run starts the specified command and waits for it to complete.

If the command runs successfully and copying streams are done as expected, the error is nil.

If the container exits with a non-zero exit code, the error is of type *ExitError. Other error types may be returned for I/O problems and such.

func (*Cmd) Start

func (c *Cmd) Start() error

Start starts the specified command but does not wait for it to complete.

func (*Cmd) StderrPipe

func (c *Cmd) StderrPipe() (io.ReadCloser, error)

StderrPipe returns a pipe that will be connected to the command's standard error when the command starts.

Wait will close the pipe after seeing the command exit or in error conditions.

func (*Cmd) StdinPipe

func (c *Cmd) StdinPipe() (io.WriteCloser, error)

StdinPipe returns a pipe that will be connected to the command's standard input when the command starts.

Different than os/exec.StdinPipe, returned io.WriteCloser should be closed by user.

func (*Cmd) StdoutPipe

func (c *Cmd) StdoutPipe() (io.ReadCloser, error)

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

Wait will close the pipe after seeing the command exit or in error conditions.

func (*Cmd) Wait

func (c *Cmd) Wait() error

Wait waits for the command to exit. It must have been started by Start.

If the container exits with a non-zero exit code, the error is of type *ExitError. Other error types may be returned for I/O problems and such.

Different than os/exec.Wait, this method will not release any resources associated with Cmd (such as file handles).

type Docker

type Docker struct {
	*docker.Client
}

Docker contains connection to Docker API. Use github.com/fsouza/go-dockerclient to initialize *docker.Client.

func (Docker) Command

func (d Docker) Command(method Execution, name string, arg ...string) *Cmd

Command returns the Cmd struct to execute the named program with given arguments using specified execution method.

For each new Cmd, you should create a new instance for "method" argument.

type Execution

type Execution interface {
	// contains filtered or unexported methods
}

Execution determines how the command is going to be executed. Currently the only method is ByCreatingContainer.

func ByCreatingContainer

func ByCreatingContainer(opts docker.CreateContainerOptions) (Execution, error)

ByCreatingContainer is the execution strategy where a new container with specified options is created to execute the command.

The container will be created and started with Cmd.Start and will be deleted before Cmd.Wait returns.

type ExitError

type ExitError struct {
	// ExitCode holds the non-zero exit code of the container
	ExitCode int

	// Stderr holds the standard error output from the command
	// if it *Cmd executed through Output() and Cmd.Stderr was not
	// set.
	Stderr []byte
}

ExitError reports an unsuccessful exit by a command.

func (*ExitError) Error

func (e *ExitError) Error() string