libcontainer

README

libcontainer - reference implementation for containers

Libcontainer provides a native Go implementation for creating containers with namespaces, cgroups, capabilities, and filesystem access controls. It allows you to manage the lifecycle of the container performing additional operations after the container is created.

Container

A container is a self contained execution environment that shares the kernel of the host system and which is (optionally) isolated from other containers in the system.

Using libcontainer

To create a container you first have to initialize an instance of a factory that will handle the creation and initialization for a container.

Because containers are spawned in a two step process you will need to provide arguments to a binary that will be executed as the init process for the container. To use the current binary that is spawning the containers and acting as the parent you can use os.Args[0] and we have a command called init setup.

root, err := libcontainer.New("/var/lib/container", libcontainer.InitArgs(os.Args[0], "init"))
if err != nil {
    log.Fatal(err)
}

Once you have an instance of the factory created we can create a configuration struct describing how the container is to be created. A sample would look similar to this:

config := &configs.Config{
    Rootfs: rootfs,
    Capabilities: []string{
        "CHOWN",
        "DAC_OVERRIDE",
        "FSETID",
        "FOWNER",
        "MKNOD",
        "NET_RAW",
        "SETGID",
        "SETUID",
        "SETFCAP",
        "SETPCAP",
        "NET_BIND_SERVICE",
        "SYS_CHROOT",
        "KILL",
        "AUDIT_WRITE",
    },
    Namespaces: configs.Namespaces([]configs.Namespace{
        {Type: configs.NEWNS},
        {Type: configs.NEWUTS},
        {Type: configs.NEWIPC},
        {Type: configs.NEWPID},
        {Type: configs.NEWNET},
    }),
    Cgroups: &configs.Cgroup{
        Name:            "test-container",
        Parent:          "system",
        AllowAllDevices: false,
        AllowedDevices:  configs.DefaultAllowedDevices,
    },

    Devices:  configs.DefaultAutoCreatedDevices,
    Hostname: "testing",
    Networks: []*configs.Network{
        {
            Type:    "loopback",
            Address: "127.0.0.1/0",
            Gateway: "localhost",
        },
    },
    Rlimits: []configs.Rlimit{
        {
            Type: syscall.RLIMIT_NOFILE,
            Hard: uint64(1024),
            Soft: uint64(1024),
        },
    },
}

Once you have the configuration populated you can create a container:

container, err := root.Create("container-id", config)

To spawn bash as the initial process inside the container and have the processes pid returned in order to wait, signal, or kill the process:

process := &libcontainer.Process{
    Args:   []string{"/bin/bash"},
    Env:    []string{"PATH=/bin"},
    User:   "daemon",
    Stdin:  os.Stdin,
    Stdout: os.Stdout,
    Stderr: os.Stderr,
}

err := container.Start(process)
if err != nil {
    log.Fatal(err)
}

// wait for the process to finish.
status, err := process.Wait()
if err != nil {
    log.Fatal(err)
}

// destroy the container.
container.Destroy()

Additional ways to interact with a running container are:

// return all the pids for all processes running inside the container.
processes, err := container.Processes()

// get detailed cpu, memory, io, and network statistics for the container and
// it's processes.
stats, err := container.Stats()


// pause all processes inside the container.
container.Pause()

// resume all paused processes.
container.Resume()

nsinit

nsinit is a cli application which demonstrates the use of libcontainer. It is able to spawn new containers or join existing containers. A root filesystem must be provided for use along with a container configuration file.

To build nsinit, run make binary. It will save the binary into bundles/nsinit.

To use nsinit, cd into a Linux rootfs and copy a container.json file into the directory with your specified configuration. Environment, networking, and different capabilities for the container are specified in this file. The configuration is used for each process executed inside the container.

See the sample_configs folder for examples of what the container configuration should look like.

To execute /bin/bash in the current directory as a container just run the following as root:

nsinit exec --tty /bin/bash

If you wish to spawn another process inside the container while your current bash session is running, run the same command again to get another bash shell (or change the command). If the original process (PID 1) dies, all other processes spawned inside the container will be killed and the namespace will be removed.

You can identify if a process is running in a container by looking to see if state.json is in the root of the directory.

You may also specify an alternate root place where the container.json file is read and where the state.json file will be saved.

Checkpoint & Restore

libcontainer now integrates CRIU for checkpointing and restoring containers. This let's you save the state of a process running inside a container to disk, and then restore that state into a new process, on the same machine or on another machine.

criu version 1.5.2 or higher is required to use checkpoint and restore. If you don't already have criu installed, you can build it from source, following the online instructions. criu is also installed in the docker image generated when building libcontainer with docker.

To try an example with nsinit, open two terminals to the same busybox directory. In the first terminal, run a command like this one:

nsinit exec -- sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

You should see logs printing to the terminal every second. Now, in the second terminal, run:

nsinit checkpoint --image-path=/tmp/criu

The logs in your first terminal will stop and the process will exit. Finally, in the second terminal, run the restore command:

nsinit restore --image-path=/tmp/criu

The process will resume counting where it left off and printing to the new terminal window.

Future

See the roadmap.

Copyright and license

Code and documentation copyright 2014 Docker, inc. Code released under the Apache 2.0 license. Docs released under Creative commons.

Hacking on libcontainer

First of all, please familiarise yourself with the libcontainer Principles.

If you're a contributor or aspiring contributor, you should read the Contributors' Guide.

If you're a maintainer or aspiring maintainer, you should read the Maintainers' Guide and "How can I become a maintainer?" in the Contributors' Guide.

Documentation

Overview

Libcontainer provides a native Go implementation for creating containers with namespaces, cgroups, capabilities, and filesystem access controls. It allows you to manage the lifecycle of the container performing additional operations after the container is created.

Index

• func Cgroupfs(l *LinuxFactory) error
• func InitArgs(args ...string) func(*LinuxFactory) error
• func InitPath(path string, args ...string) func(*LinuxFactory) error
• func SystemdCgroups(l *LinuxFactory) error
• func TmpfsRoot(l *LinuxFactory) error
• type Console
• type Container
• type CriuOpts
• type CriuPageServerInfo
• type Error
• type ErrorCode
  • func (c ErrorCode) String() string
• type Factory
  • func New(root string, options ...func(*LinuxFactory) error) (Factory, error)
• type LinuxFactory
  • func (l *LinuxFactory) Create(id string, config *configs.Config) (Container, error)
  • func (l *LinuxFactory) Load(id string) (Container, error)
  • func (l *LinuxFactory) StartInitialization() (err error)
  • func (l *LinuxFactory) Type() string
• type NetworkInterface
• type Process
  • func (p *Process) NewConsole(rootuid int) (Console, error)
  • func (p Process) Pid() (int, error)
  • func (p Process) Signal(sig os.Signal) error
  • func (p Process) Wait() (*os.ProcessState, error)
• type State
• type Stats
• type Status

Functions

func Cgroupfs

func Cgroupfs(l *LinuxFactory) error

Cgroupfs is an options func to configure a LinuxFactory to return containers that use the native cgroups filesystem implementation to create and manage cgroups.

func InitArgs

func InitArgs(args ...string) func(*LinuxFactory) error

InitArgs returns an options func to configure a LinuxFactory with the provided init arguments.

func InitPath

func InitPath(path string, args ...string) func(*LinuxFactory) error

InitPath returns an options func to configure a LinuxFactory with the provided absolute path to the init binary and arguements.

func SystemdCgroups

func SystemdCgroups(l *LinuxFactory) error

SystemdCgroups is an options func to configure a LinuxFactory to return containers that use systemd to create and manage cgroups.

func TmpfsRoot

func TmpfsRoot(l *LinuxFactory) error

TmpfsRoot is an option func to mount LinuxFactory.Root to tmpfs.

Types

type Console

type Console interface {
	io.ReadWriter
	io.Closer

	// Path returns the filesystem path to the slave side of the pty.
	Path() string

	// Fd returns the fd for the master of the pty.
	Fd() uintptr
}

Console represents a pseudo TTY.

type Container

type Container interface {
	// Returns the ID of the container
	ID() string

	// Returns the current status of the container.
	//
	// errors:
	// ContainerDestroyed - Container no longer exists,
	// Systemerror - System error.
	Status() (Status, error)

	// State returns the current container's state information.
	//
	// errors:
	// Systemerror - System error.
	State() (*State, error)

	// Returns the current config of the container.
	Config() configs.Config

	// Returns the PIDs inside this container. The PIDs are in the namespace of the calling process.
	//
	// errors:
	// ContainerDestroyed - Container no longer exists,
	// Systemerror - System error.
	//
	// Some of the returned PIDs may no longer refer to processes in the Container, unless
	// the Container state is PAUSED in which case every PID in the slice is valid.
	Processes() ([]int, error)

	// Returns statistics for the container.
	//
	// errors:
	// ContainerDestroyed - Container no longer exists,
	// Systemerror - System error.
	Stats() (*Stats, error)

	// Set cgroup resources of container as configured
	//
	// We can use this to change resources when containers are running.
	//
	// errors:
	// Systemerror - System error.
	Set(config configs.Config) error

	// Start a process inside the container. Returns error if process fails to
	// start. You can track process lifecycle with passed Process structure.
	//
	// errors:
	// ContainerDestroyed - Container no longer exists,
	// ConfigInvalid - config is invalid,
	// ContainerPaused - Container is paused,
	// Systemerror - System error.
	Start(process *Process) (err error)

	// Checkpoint checkpoints the running container's state to disk using the criu(8) utility.
	//
	// errors:
	// Systemerror - System error.
	Checkpoint(criuOpts *CriuOpts) error

	// Restore restores the checkpointed container to a running state using the criu(8) utiity.
	//
	// errors:
	// Systemerror - System error.
	Restore(process *Process, criuOpts *CriuOpts) error

	// Destroys the container after killing all running processes.
	//
	// Any event registrations are removed before the container is destroyed.
	// No error is returned if the container is already destroyed.
	//
	// errors:
	// Systemerror - System error.
	Destroy() error

	// If the Container state is RUNNING or PAUSING, sets the Container state to PAUSING and pauses
	// the execution of any user processes. Asynchronously, when the container finished being paused the
	// state is changed to PAUSED.
	// If the Container state is PAUSED, do nothing.
	//
	// errors:
	// ContainerDestroyed - Container no longer exists,
	// Systemerror - System error.
	Pause() error

	// If the Container state is PAUSED, resumes the execution of any user processes in the
	// Container before setting the Container state to RUNNING.
	// If the Container state is RUNNING, do nothing.
	//
	// errors:
	// ContainerDestroyed - Container no longer exists,
	// Systemerror - System error.
	Resume() error

	// NotifyOOM returns a read-only channel signaling when the container receives an OOM notification.
	//
	// errors:
	// Systemerror - System error.
	NotifyOOM() (<-chan struct{}, error)
}

A libcontainer container object.

Each container is thread-safe within the same process. Since a container can be destroyed by a separate process, any function may return that the container was not found.

type CriuOpts

type CriuOpts struct {
	ImagesDirectory         string             // directory for storing image files
	WorkDirectory           string             // directory to cd and write logs/pidfiles/stats to
	LeaveRunning            bool               // leave container in running state after checkpoint
	TcpEstablished          bool               // checkpoint/restore established TCP connections
	ExternalUnixConnections bool               // allow external unix connections
	ShellJob                bool               // allow to dump and restore shell jobs
	PageServer              CriuPageServerInfo // allow to dump to criu page server
}

type CriuPageServerInfo

type CriuPageServerInfo struct {
	Address string // IP address of CRIU page server
	Port    int32  // port number of CRIU page server
}

type Error

type Error interface {
	error

	// Returns a verbose string including the error message
	// and a representation of the stack trace suitable for
	// printing.
	Detail(w io.Writer) error

	// Returns the error code for this error.
	Code() ErrorCode
}

API Error type.

type ErrorCode

type ErrorCode int

API error code type.

const (
	// Factory errors
	IdInUse ErrorCode = iota
	InvalidIdFormat

	// Container errors
	ContainerNotExists
	ContainerPaused
	ContainerNotStopped
	ContainerNotRunning

	// Process errors
	ProcessNotExecuted

	// Common errors
	ConfigInvalid
	SystemError
)

API error codes.

func (ErrorCode) String

func (c ErrorCode) String() string

type Factory

type Factory interface {
	// Creates a new container with the given id and starts the initial process inside it.
	// id must be a string containing only letters, digits and underscores and must contain
	// between 1 and 1024 characters, inclusive.
	//
	// The id must not already be in use by an existing container. Containers created using
	// a factory with the same path (and file system) must have distinct ids.
	//
	// Returns the new container with a running process.
	//
	// errors:
	// IdInUse - id is already in use by a container
	// InvalidIdFormat - id has incorrect format
	// ConfigInvalid - config is invalid
	// Systemerror - System error
	//
	// On error, any partially created container parts are cleaned up (the operation is atomic).
	Create(id string, config *configs.Config) (Container, error)

	// Load takes an ID for an existing container and returns the container information
	// from the state.  This presents a read only view of the container.
	//
	// errors:
	// Path does not exist
	// Container is stopped
	// System error
	Load(id string) (Container, error)

	// StartInitialization is an internal API to libcontainer used during the reexec of the
	// container.
	//
	// Errors:
	// Pipe connection error
	// System error
	StartInitialization() error

	// Type returns info string about factory type (e.g. lxc, libcontainer...)
	Type() string
}

func New

func New(root string, options ...func(*LinuxFactory) error) (Factory, error)

New returns a linux based container factory based in the root directory and configures the factory with the provided option funcs.

type LinuxFactory

type LinuxFactory struct {
	// Root directory for the factory to store state.
	Root string

	// InitPath is the absolute path to the init binary.
	InitPath string

	// InitArgs are arguments for calling the init responsibilities for spawning
	// a container.
	InitArgs []string

	// CriuPath is the path to the criu binary used for checkpoint and restore of
	// containers.
	CriuPath string

	// Validator provides validation to container configurations.
	Validator validate.Validator

	// NewCgroupsManager returns an initialized cgroups manager for a single container.
	NewCgroupsManager func(config *configs.Cgroup, paths map[string]string) cgroups.Manager
}

LinuxFactory implements the default factory interface for linux based systems.

func (*LinuxFactory) Create

func (l *LinuxFactory) Create(id string, config *configs.Config) (Container, error)

func (*LinuxFactory) Load

func (l *LinuxFactory) Load(id string) (Container, error)

func (*LinuxFactory) StartInitialization

func (l *LinuxFactory) StartInitialization() (err error)

StartInitialization loads a container by opening the pipe fd from the parent to read the configuration and state This is a low level implementation detail of the reexec and should not be consumed externally

func (*LinuxFactory) Type

func (l *LinuxFactory) Type() string

type NetworkInterface

type NetworkInterface struct {
	// Name is the name of the network interface.
	Name string

	RxBytes   uint64
	RxPackets uint64
	RxErrors  uint64
	RxDropped uint64
	TxBytes   uint64
	TxPackets uint64
	TxErrors  uint64
	TxDropped uint64
}

type Process

type Process struct {
	// The command to be run followed by any arguments.
	Args []string

	// Env specifies the environment variables for the process.
	Env []string

	// User will set the uid and gid of the executing process running inside the container
	// local to the container's user and group configuration.
	User string

	// Cwd will change the processes current working directory inside the container's rootfs.
	Cwd string

	// Stdin is a pointer to a reader which provides the standard input stream.
	Stdin io.Reader

	// Stdout is a pointer to a writer which receives the standard output stream.
	Stdout io.Writer

	// Stderr is a pointer to a writer which receives the standard error stream.
	Stderr io.Writer

	// ExtraFiles specifies additional open files to be inherited by the container
	ExtraFiles []*os.File

	// Capabilities specify the capabilities to keep when executing the process inside the container
	// All capabilities not specified will be dropped from the processes capability mask
	Capabilities []string
	// contains filtered or unexported fields
}

Process specifies the configuration and IO for a process inside a container.

func (*Process) NewConsole

func (p *Process) NewConsole(rootuid int) (Console, error)

NewConsole creates new console for process and returns it

func (Process) Pid

func (p Process) Pid() (int, error)

Pid returns the process ID

func (Process) Signal

func (p Process) Signal(sig os.Signal) error

Signal sends a signal to the Process.

func (Process) Wait

func (p Process) Wait() (*os.ProcessState, error)

Wait waits for the process to exit. Wait releases any resources associated with the Process

type State

type State struct {
	// ID is the container ID.
	ID string `json:"id"`

	// InitProcessPid is the init process id in the parent namespace.
	InitProcessPid int `json:"init_process_pid"`

	// InitProcessStartTime is the init process start time.
	InitProcessStartTime string `json:"init_process_start"`

	// Path to all the cgroups setup for a container. Key is cgroup subsystem name
	// with the value as the path.
	CgroupPaths map[string]string `json:"cgroup_paths"`

	// NamespacePaths are filepaths to the container's namespaces. Key is the namespace type
	// with the value as the path.
	NamespacePaths map[configs.NamespaceType]string `json:"namespace_paths"`

	// Config is the container's configuration.
	Config configs.Config `json:"config"`

	// Container's standard descriptors (std{in,out,err}), needed for checkpoint and restore
	ExternalDescriptors []string `json:"external_descriptors,omitempty"`
}

State represents a running container's state

type Stats

type Stats struct {
	Interfaces  []*NetworkInterface
	CgroupStats *cgroups.Stats
}

type Status

type Status int

The status of a container.

const (
	// The container exists and is running.
	Running Status = iota + 1

	// The container exists, it is in the process of being paused.
	Pausing

	// The container exists, but all its processes are paused.
	Paused

	// The container exists, but its state is saved on disk
	Checkpointed

	// The container does not exist.
	Destroyed
)