oci

README

internal/pkg/runtime/launcher/oci

This package contains routines that configure and launch a container in an OCI bundle format, using a low-level OCI runtime, either crun or runc at this time. crun is currently preferred. runc is used where crun is not available.

Note - at present, all functionality works with either crun or runc. However, in future crun may be required for all functionality, as runc does not support some limited ID mappings etc. that may be beneficial in an HPC scenario.

The package contrasts with internal/pkg/runtime/launcher/native which executes Singularity format containers (SIF/Sandbox/squashfs/ext3), using one of our own runtime engines (internal/pkg/runtime/engine/*).

There are two flows that are implemented here.

• Basic OCI runtime operations against an existing bundle, which will be executed via the singularity oci command group. These are not widely used by end-users of singularity.
• A Launcher, that implements an Exec function that will be called by 'actions' (run/shell/exec) in --oci mode, and will:
  • Prepare an OCI bundle according to launcher.Options passed through from the CLI layer.
  • Execute the bundle, interactively, via the OCI Run operation.

Note - this area of code is under heavy development for experimental introduction in CE 3.11. It is likely that it will be heavily refactored, and split, in future.

Basic OCI Operations

The following files implement basic OCI operations on a runtime bundle:

oci_linux.go

Defines constants, path resolution, and minimal bundle locking functions.

oci_runc_linux.go

Holds implementations of the Run / Start / Exec / Kill / Delete / Pause / Resume / State OCI runtime operations.

See https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#operations

These functions are thin wrappers around the runc/crun operations of the same name.

oci_conmon_linux.go

Hold an implementation of the Create OCI runtime operation. This calls out to conmon, which in turn calls crun or runc.

conmon is used to manage logging and console streams for containers that are started backgrounded, so we don't have to do that ourselves.

oci_attach_linux.go

Implements an Attach function, which can attach the user's console to the streams of a container running in the background, which is being monitored by conmon.

Testing

End-to-end flows of basic OCI operations on an existing bundle are tested in the OCI group of the e2e suite, e2e/oci.

Launcher Flow

The Launcher type connects the standard singularity CLI actions (run/shell/exec), to execution of an OCI container in a native bundle. Invoked with the --oci flag, this is in contrast to running a Singularity format container, with Singularity's own runtime engine.

spec_linux.go

Provides a minimal OCI runtime spec, that will form the basis of container execution that is roughly comparable to running a native singularity container with --compat (--containall).

mounts_linux.go

Provides code handling the addition of required mounts to the OCI runtime spec.

process_linux.go

Provides code handling configuration of container process execution, including user mapping.

launcher_linux.go

Implements Launcher.Exec, which is called from the CLI layer. It will:

• Create a temporary bundle directory.
• Use pkg/ocibundle/native to retrieve the specified image, and extract it in the temporary bundle.
• Configure the container by creating an appropriate runtime spec.
• Call the interactive OCI Run function to execute the container with crun or runc.

Namespace Considerations

An OCI container started via Launch.Exec as a non-root user always uses at least one user namespace.

The user namespace is created prior to calling runc or crun, so we'll call it an outer user namespace.

Creation of this outer user namespace is via using the RunNS function, instead of Run. The RunNS function executes the Singularity starter binary, with a minimal configuration of the fakeroot engine ( internal/pkg/runtime/engine/fakeroot/config).

The starter will create a user namespace and ID mapping, and will then execute singularity oci run to perform the basic OCI Run operation against the bundle that the Launcher.Exec function has prepared.

The outer user namespace from which runc or crun is called always maps the host user id to root inside the userns.

When a container is run in --fakeroot mode, the outer user namespace is the only user namespace. The OCI runtime config does not request any additional userns or ID mapping be performed by crun / runc.

When a container is not run in --fakeroot mode, the OCI runtime config for the bundle requests that crun / runc:

• Create another, inner, user namespace for the container.
• Apply an ID mapping which reverses the 'fakeroot' outer ID mapping.

I.E. when a container runs without --fakeroot, the ID mapping is:

• User ID on host (1001)
• Root in outer user namespace (0)
• User ID in container (1001)

Testing

End-to-end testing of ßthe launcher flow is via the e2e/actions suite. Tests prefixed oci.

Documentation

Overview

Package oci implements a Launcher that will configure and launch a container with an OCI runtime. It also provides implementations of OCI state transitions that can be called directly, Create/Start/Kill etc.

Package oci implements a Launcher that will configure and launch a container with an OCI runtime. It also provides implementations of OCI state transitions that can be called directly, Create/Start/Kill etc.

Index

• Constants
• Variables
• func Attach(containerID string) error
• func Create(containerID, bundlePath string, systemdCgroups bool) error
• func CrunNestCgroup() error
• func Delete(ctx context.Context, containerID string, systemdCgroups bool) error
• func Exec(containerID string, cmdArgs []string, systemdCgroups bool) error
• func Kill(containerID string, killSignal string) error
• func Pause(containerID string, systemdCgroups bool) error
• func Resume(containerID string, systemdCgroups bool) error
• func Run(ctx context.Context, containerID, bundlePath, pidFile string, ...) error
• func Runtime() (path string, err error)
• func Start(containerID string, systemdCgroups bool) error
• func State(containerID string, systemdCgroups bool) error
• func Update(containerID, cgFile string, systemdCgroups bool) error
• func WrapWithWritableTmpFs(ctx context.Context, f func() error, bundleDir string, allowSetuid bool) error
• type Launcher
  • func NewLauncher(opts ...launcher.Option) (*Launcher, error)
  • func (l *Launcher) Exec(ctx context.Context, ep launcher.ExecParams) error
  • func (l *Launcher) RunWrapped(ctx context.Context, containerID, bundlePath, pidFile string) error
  • func (l *Launcher) WrapWithOverlays(ctx context.Context, f func() error, bundleDir string) error

Constants

const (
	AttachPipeStdin  = 1
	AttachPipeStdout = 2
	AttachPipeStderr = 3
)

Sync with stdpipe_t in conmon.c

const DetachKeys = "ctrl-p,ctrl-q"

DetachKeys is the key sequence for detaching a container.

Variables

var (
	ErrUnsupportedOption = errors.New("not supported by OCI launcher")
	ErrNotImplemented    = errors.New("not implemented by OCI launcher")
)

var ErrDetach = errors.New("detached from container")

Functions

func Attach

func Attach(containerID string) error

Attach attaches the console to a running container

func Create

func Create(containerID, bundlePath string, systemdCgroups bool) error

Create creates a container from an OCI bundle

func CrunNestCgroup

func CrunNestCgroup() error

crunNestCgroup will check whether we are using crun, and enter a cgroup if running as a non-root user under cgroups v2, with systemd. This is required to satisfy a common user-owned ancestor cgroup requirement on e.g. bare ssh logins. See: https://github.com/sylabs/singularity/issues/1538

func Delete

func Delete(ctx context.Context, containerID string, systemdCgroups bool) error

Delete deletes container resources

func Exec

func Exec(containerID string, cmdArgs []string, systemdCgroups bool) error

Exec executes a command in a container

func Kill

func Kill(containerID string, killSignal string) error

Kill kills container process

func Pause

func Pause(containerID string, systemdCgroups bool) error

Pause pauses processes in a container

func Resume

func Resume(containerID string, systemdCgroups bool) error

Resume un-pauses processes in a container

func Run

func Run(ctx context.Context, containerID, bundlePath, pidFile string, systemdCgroups bool) error

Run runs a container (equivalent to create/start/delete)

func Runtime

func Runtime() (path string, err error)

Runtime returns path to the OCI Runtime - crun (preferred), or runc.

func Start

func Start(containerID string, systemdCgroups bool) error

Start starts a previously created container

func State

func State(containerID string, systemdCgroups bool) error

State queries container state

func Update

func Update(containerID, cgFile string, systemdCgroups bool) error

Update updates container cgroups resources

func WrapWithWritableTmpFs

func WrapWithWritableTmpFs(ctx context.Context, f func() error, bundleDir string, allowSetuid bool) error

WrapWithWritableTmpFs runs a function wrapped with prep / cleanup steps for a tmpfs. This tmpfs is always writable so that the launcher and runtime are able to add content to the container. Whether it is writable from inside the container is controlled by the runtime config.

Types

type Launcher

type Launcher struct {
	// contains filtered or unexported fields
}

Launcher will holds configuration for, and will launch a container using an OCI runtime.

func NewLauncher

func NewLauncher(opts ...launcher.Option) (*Launcher, error)

NewLauncher returns a oci.Launcher with an initial configuration set by opts.

func (*Launcher) Exec

func (l *Launcher) Exec(ctx context.Context, ep launcher.ExecParams) error

Exec will interactively execute a container via the runc low-level runtime. image is a reference to an OCI image, e.g. docker://ubuntu or oci:/tmp/mycontainer

func (*Launcher) RunWrapped

func (l *Launcher) RunWrapped(ctx context.Context, containerID, bundlePath, pidFile string) error

RunWrapped runs a container via the OCI runtime, wrapped with prep / cleanup steps.

func (*Launcher) WrapWithOverlays

func (l *Launcher) WrapWithOverlays(ctx context.Context, f func() error, bundleDir string) error

WrapWithOverlays runs a function wrapped with prep / cleanup steps for the overlays in the image, and/or specified in overlayPaths. If there is no writable overlay, it adds an ephemeral overlay which is always writable so that the launcher and runtime are able to add content to the container. Whether an ephemeral overlay is writable from inside the container is controlled by the runtime config.