docker

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 30, 2024 License: Apache-2.0 Imports: 33 Imported by: 0

README

docker

Table of Contents

  1. Description
  2. Structure and Organisation
  3. Class Diagram
  4. Functionality
  5. Data Types
  6. Testing
  7. Proposed Functionality/Requirements
  8. References

Specification

Description

This sub-package contains functionality including drivers and api for the Docker executor.

Structure and Organisation

Here is quick overview of the contents of this pacakge:

  • README: Current file which is aimed towards developers who wish to use and modify the docker functionality.

  • client: This file provides a high level wrapper around the docker library.

  • executor: This is the main implementation of the executor interface for docker. It is the entry point of the sub-package. It is intended to be used as a singleton.

  • handler: This file contains a handler implementation to manage the lifecycle of a single job.

  • init: This file is responsible for initialization of the package. Currently it only initializes a logger to be used through out the sub-package.

  • types: This file contains Models that are specifically related to the docker executor. Mainly it contains the engine spec model that describes a docker job.

Files with *_test.go suffix contain unit tests for the functionality in corresponding file.

Class Diagram
Source

docker class diagram

Rendered from source file
!$rootUrlGitlab = "https://gitlab.com/nunet/device-management-service/-/raw/main"
!$packageRelativePath = "/executor/docker"
!$packageUrlGitlab = $rootUrlGitlab + $packageRelativePath
 
!include $packageUrlGitlab/specs/class_diagram.puml
Functionality

Below methods have been implemented in this package:

NewExecutor

  • signature: NewExecutor(ctx context.Context, id string) -> (executor.docker.Executor, error)

  • input #1: Go context

  • input #2: identifier of the executor

  • output (sucess): Executor instance of type executor.docker.Executor

  • output (error): error

NewExecutor function initializes a new Executor instance with a Docker client. It returns an error if Docker client initialization fails.

It is expecte that NewExecutor would be called prior to calling any other executor functions. The Executor instance returned would then be used to call other functions like Start, Stop etc.

Start

For function signature refer to the package readme

Start function begins the execution of a request by starting a Docker container. It creates the container based on the configuration parameters provided in the execution request. It returns an error message if

  • container is already started
  • container execution is finished
  • there is failure is creation of a new container
Wait

For function signature refer to the package readme

Wait initiates a wait for the completion of a specific execution using its executionID. The function returns two channels: one for the result and another for any potential error.

If the executionID is not found, an error is immediately sent to the error channel.

Otherwise, an internal goroutine is spawned to handle the asynchronous waiting. The entity calling should use the two returned channels to wait for the result of the execution or an error. If there is a cancellation request (context is done) before completion, an error is relayed to the error channel. When the execution is finished, both the channels are closed.

Cancel

For function signature refer to the package readme

Cancel tries to terminate an ongoing execution identified by its executionID. It returns an error if the execution does not exist.

GetLogStream

For function signature refer to the package readme

GetLogStream provides a stream of output logs for a specific execution. Parameters tail and follow specified in executor.LogStreamRequest provided as input control whether to include past logs and whether to keep the stream open for new logs, respectively.

It returns an error if the execution is not found.

Run

For function signature refer to the package readme

Run initiates and waits for the completion of an execution in one call. This method serves as a higher-level convenience function that internally calls Start and Wait methods. It returns the result of the execution as executor.ExecutionResult type.

It returns an error in case of:

  • failure in starting the container
  • failure in waiting
  • context is cancelled
ConfigureHostConfig

  • signature: configureHostConfig(vendor types.GPUVendor, params *types.ExecutionRequest, mounts []mount.Mount) container.HostConfig

  • input #1: GPU vendor (types.GPUVendor)

  • input #2: Execution request parameters (types.ExecutionRequest)

  • input #3: List of container mounts ([]mount.Mount)

  • output: Host configuration for the Docker container (container.HostConfig)

The configureHostConfig function sets up the host configuration for the container based on the GPU vendor and resources requested by the execution. It supports configurations for different types of GPUs and CPUs.

The function performs the following steps:

  1. NVIDIA GPUs:

    • Configures the DeviceRequests to include all GPUs specified in the execution request.
    • Sets the memory and CPU resources according to the request parameters.

  2. AMD GPUs:

    • Binds the necessary device paths (/dev/kfd and /dev/dri) to the container.
    • Adds the video group to the container.
    • Sets the memory and CPU resources according to the request parameters.

  3. Intel GPUs:

    • Binds the /dev/dri directory to the container, exposing all Intel GPUs.
    • Sets the memory and CPU resources according to the request parameters.

  4. Default (CPU-only):

    • Configures the container with memory and CPU resources only, without any GPU-specific settings.

The function ensures that the appropriate resources and device paths are allocated to the container based on the available and requested GPU resources.

Cleanup

  • signature: Cleanup(ctx context.Context) -> error

  • input: Go context

  • output (sucess): None

  • output (error): error

Cleanup removes all Docker resources associated with the executor. This includes removing containers including networks and volumes with the executor's label. It returns an error it if unable to remove the containers.

Data Types
  • executor.docker.Executor: This is the instance of the executor created by NewExecutor function. It contains the Docker client and other resources required to execute requests.
import (
	"sync"

	"github.com/docker/docker/client"
)

// Executor manages the lifecycle of Docker containers for execution requests
type Executor struct {
	// ID is the identifier of the executor instance
	ID string

	// handlers maps execution IDs to their handlers
	handlers SyncMap[string, executionHandler]

	// client embeds Docker client for container management
	client Client
}

// Client wraps the Docker client to provide high-level operations on Docker containers and networks
type Client struct {
	// client embeds the Docker client
	client *client.Client
}

// A SyncMap is a concurrency-safe sync.Map that uses strongly-typed
// method signatures to ensure the types of its stored data are known.
type SyncMap[K comparable, V any] struct {
	sync.Map
}
  • executor.docker.executionHandler: This contains necessary information to manage the execution of a docker container.
// executionHandler manages the lifecycle and execution of a Docker container for a specific job.
type executionHandler struct {
	// provided by the executor
	ID string

	client Client // Docker client for container management.

	// meta data about the task
	jobID       string
	executionID string
	containerID string
	// Directory to store execution results
	resultsDir  string 

	// synchronization
	activeCh chan bool    // Blocks until the container starts running.
	waitCh   chan bool    // BLocks until execution completes or fails.
	running  *atomic.Bool // Indicates if the container is currently running.

	// result of the execution
	result executor.ExecutionResult
}

Refer to package readme for other data types.

Testing

Unit tests for each functionality are defined in files with *_test.go naming convention.

Proposed Functionality / Requirements
List of issues

All issues that are related to the implementation of executor package can be found below. These include any proposals for modifications to the package or new functionality needed to cover the requirements of other packages.

References

Documentation

Index

Constants

View Source 
const (
	EngineKeyImage            = "image"
	EngineKeyEntrypoint       = "entrypoint"
	EngineKeyCmd              = "cmd"
	EngineKeyEnvironment      = "environment"
	EngineKeyWorkingDirectory = "working_directory"
)

Variables

View Source 
var DestroyTimeout = time.Second * 10
View Source 
var ErrNotInstalled = errors.New("docker is not installed")

Functions

This section is empty.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client wraps the Docker client to provide high-level operations on Docker containers and networks.

func NewDockerClient 

func NewDockerClient() (*Client, error)

NewDockerClient initializes a new Docker client with environment variables and API version negotiation.

func (*Client) CreateContainer 

func (c *Client) CreateContainer(
	ctx context.Context,
	config *container.Config,
	hostConfig *container.HostConfig,
	networkingConfig *network.NetworkingConfig,
	platform *v1.Platform,
	name string,
	pullImage bool,
) (string, error)

CreateContainer creates a new Docker container with the specified configuration.

func (*Client) Exec 

func (c *Client) Exec(ctx context.Context, containerID string, cmd []string) (int, string, string, error)

Exec executes a command inside a running container. Returns (exit code, stdout, stderr, and an error if the operation fails)

func (*Client) FindContainer 

func (c *Client) FindContainer(ctx context.Context, label string, value string) (string, error)

FindContainer searches for a container by label and value, returning its ID if found.

func (*Client) FollowLogs 

func (c *Client) FollowLogs(ctx context.Context, id string) (stdout, stderr io.Reader, err error)

FollowLogs tails the logs of a specified container, returning separate readers for stdout and stderr.

func (*Client) GetImage 

func (c *Client) GetImage(ctx context.Context, imageName string) (image.Summary, error)

GetImage returns detailed information about a Docker image.

func (*Client) GetOutputStream 

func (c *Client) GetOutputStream(
	ctx context.Context,
	containerID string,
	since string,
	follow bool,
) (io.ReadCloser, error)

GetOutputStream streams the logs for a specified container. The 'since' parameter specifies the timestamp from which to start streaming logs. The 'follow' parameter indicates whether to continue streaming logs as they are produced. Returns an io.ReadCloser to read the output stream and an error if the operation fails.

func (*Client) HasImage 

func (c *Client) HasImage(ctx context.Context, imageName string) bool

HasImage checks if an image exists locally

func (*Client) InspectContainer 

func (c *Client) InspectContainer(ctx context.Context, id string) (types.ContainerJSON, error)

InspectContainer returns detailed information about a Docker container.

func (*Client) IsInstalled 

func (c *Client) IsInstalled(ctx context.Context) bool

IsInstalled checks if Docker is installed and reachable by pinging the Docker daemon.

func (*Client) PauseContainer 

func (c *Client) PauseContainer(ctx context.Context, containerID string) error

PauseContainer pauses the main process of the given container without terminating it.

func (*Client) PullImage 

func (c *Client) PullImage(ctx context.Context, imageName string) (string, error)

PullImage pulls a Docker image from a registry.

func (*Client) RemoveContainer 

func (c *Client) RemoveContainer(ctx context.Context, containerID string) error

RemoveContainer removes a Docker container, optionally forcing removal and removing associated volumes.

func (*Client) RemoveObjectsWithLabel 

func (c *Client) RemoveObjectsWithLabel(ctx context.Context, label string, value string) error

RemoveObjectsWithLabel removes all Docker containers and networks with a specific label.

func (*Client) ResumeContainer 

func (c *Client) ResumeContainer(ctx context.Context, containerID string) error

ResumeContainer resumes the process execution within the container

func (*Client) StartContainer 

func (c *Client) StartContainer(ctx context.Context, containerID string) error

StartContainer starts a specified Docker container.

func (*Client) StopContainer 

func (c *Client) StopContainer(
	ctx context.Context,
	containerID string,
	options container.StopOptions,
) error

StopContainer stops a running Docker container with a specified timeout.

func (*Client) WaitContainer 

func (c *Client) WaitContainer(
	ctx context.Context,
	containerID string,
) (<-chan container.WaitResponse, <-chan error)

WaitContainer waits for a container to stop, returning channels for the result and errors.

type ClientInterface 

type ClientInterface interface {
	IsInstalled(ctx context.Context) bool
	CreateContainer(
		ctx context.Context,
		config *container.Config,
		hostConfig *container.HostConfig,
		networkingConfig *network.NetworkingConfig,
		platform *v1.Platform,
		name string,
		pullImage bool,
	) (string, error)
	InspectContainer(ctx context.Context, id string) (types.ContainerJSON, error)
	FollowLogs(ctx context.Context, id string) (stdout, stderr io.Reader, err error)
	StartContainer(ctx context.Context, containerID string) error
	WaitContainer(
		ctx context.Context,
		containerID string,
	) (<-chan container.WaitResponse, <-chan error)
	PauseContainer(ctx context.Context, containerID string) error
	ResumeContainer(ctx context.Context, containerID string) error
	StopContainer(
		ctx context.Context,
		containerID string,
		options container.StopOptions,
	) error
	RemoveContainer(ctx context.Context, containerID string) error
	RemoveObjectsWithLabel(ctx context.Context, label string, value string) error
	FindContainer(ctx context.Context, label string, value string) (string, error)
	GetImage(ctx context.Context, imageName string) (image.Summary, error)
	PullImage(ctx context.Context, imageName string) (string, error)
	GetOutputStream(
		ctx context.Context,
		containerID string,
		since string,
		follow bool,
	) (io.ReadCloser, error)
	Exec(ctx context.Context, containerID string, cmd []string) (int, string, string, error)
}

type EngineBuilder 

type EngineBuilder struct {
	// contains filtered or unexported fields
}

EngineBuilder is a struct that is used for constructing an EngineSpec object specifically for Docker engines using the Builder pattern. It embeds an EngineBuilder object for handling the common builder methods.

func NewDockerEngineBuilder 

func NewDockerEngineBuilder(image string) *EngineBuilder

NewDockerEngineBuilder function initializes a new DockerEngineBuilder instance. It sets the engine type to model.EngineDocker.String() and image as per the input argument.

func (*EngineBuilder) Build 

func (b *EngineBuilder) Build() *types.SpecConfig

Build method constructs the final SpecConfig object by calling the embedded EngineBuilder's Build method.

func (*EngineBuilder) WithCmd 

func (b *EngineBuilder) WithCmd(c ...string) *EngineBuilder

WithCmd is a builder method that sets the Docker engine's Command. It returns the DockerEngineBuilder for further chaining of builder methods.

func (*EngineBuilder) WithEntrypoint 

func (b *EngineBuilder) WithEntrypoint(e ...string) *EngineBuilder

WithEntrypoint is a builder method that sets the Docker engine entrypoint. It returns the DockerEngineBuilder for further chaining of builder methods.

func (*EngineBuilder) WithEnvironment 

func (b *EngineBuilder) WithEnvironment(e ...string) *EngineBuilder

WithEnvironment is a builder method that sets the Docker engine's environment variables. It returns the DockerEngineBuilder for further chaining of builder methods.

func (*EngineBuilder) WithWorkingDirectory 

func (b *EngineBuilder) WithWorkingDirectory(w string) *EngineBuilder

WithWorkingDirectory is a builder method that sets the Docker engine's working directory. It returns the DockerEngineBuilder for further chaining of builder methods.

type EngineSpec 

type EngineSpec struct {
	// Image this should be pullable by docker
	Image string `json:"image,omitempty"`
	// Entrypoint optionally override the default entrypoint
	Entrypoint []string `json:"entrypoint,omitempty"`
	// Cmd specifies the command to run in the container
	Cmd []string `json:"cmd,omitempty"`
	// EnvironmentVariables is a slice of env to run the container with
	Environment []string `json:"environment,omitempty"`
	// WorkingDirectory inside the container
	WorkingDirectory string `json:"working_directory,omitempty"`
	// Privileged indicates whether the container should run with --privileged mode
	Privileged bool `json:"privileged,omitempty"`
}

EngineSpec contains necessary parameters to execute a docker job.

func DecodeSpec 

func DecodeSpec(spec *types.SpecConfig) (EngineSpec, error)

DecodeSpec decodes a spec config into a docker engine spec It converts the params into a docker EngineSpec struct and validates it

func (EngineSpec) Validate 

func (c EngineSpec) Validate() error

Validate checks if the engine spec is valid

type Executor 

type Executor struct {
	ID string
	// contains filtered or unexported fields
}

Executor manages the lifecycle of Docker containers for execution requests.

func NewExecutor 

func NewExecutor(ctx context.Context, fs afero.Afero, id string) (*Executor, error)

NewExecutor initializes a new Executor instance with a Docker client.

func (*Executor) Cancel 

func (e *Executor) Cancel(ctx context.Context, executionID string) error

Cancel tries to cancel a specific execution by its executionID. It returns an error if the execution is not found.

func (*Executor) Cleanup 

func (e *Executor) Cleanup(ctx context.Context) error

Cleanup removes all Docker resources associated with the executor. This includes removing containers including networks and volumes with the executor's label. It also removes all temporary directories created for init scripts.

func (*Executor) Exec 

func (e *Executor) Exec(ctx context.Context, executionID string, command []string) (int, string, string, error)

Exec executes a command in the container with the given containerID. Returns the exit code, stdout, stderr and an error if the execution fails.

func (*Executor) FindRunningContainer 

func (e *Executor) FindRunningContainer(
	ctx context.Context,
	jobID string,
	executionID string,
) (string, error)

FindRunningContainer finds the container that is running the execution with the given ID. It returns the container ID if found, or an error if the container is not found.

func (*Executor) GetID 

func (e *Executor) GetID() string

func (*Executor) GetLogStream 

func (e *Executor) GetLogStream(
	ctx context.Context,
	request types.LogStreamRequest,
) (io.ReadCloser, error)

GetLogStream provides a stream of output logs for a specific execution. Parameters 'withHistory' and 'follow' control whether to include past logs and whether to keep the stream open for new logs, respectively. It returns an error if the execution is not found.

func (*Executor) GetStatus 

func (e *Executor) GetStatus(ctx context.Context, executionID string) (types.ExecutionStatus, error)

GetStatus returns the status of the execution identified by its executionID. It returns the status of the execution and an error if the execution is not found or status is unknown.

func (*Executor) List 

func (e *Executor) List() []types.ExecutionListItem

List returns a slice of ExecutionListItem containing information about current executions. This implementation currently returns an empty list and should be updated in the future.

func (*Executor) Pause 

func (e *Executor) Pause(
	ctx context.Context,
	executionID string,
) error

Pause pauses the container

func (*Executor) Remove 

func (e *Executor) Remove(executionID string, timeout time.Duration) error

Remove removes a container identified by its executionID. It returns an error if the execution is not found.

func (*Executor) Resume 

func (e *Executor) Resume(
	ctx context.Context,
	executionID string,
) error

Resume resumes the container

func (*Executor) Run 

func (e *Executor) Run(
	ctx context.Context, request *types.ExecutionRequest,
) (*types.ExecutionResult, error)

Run initiates and waits for the completion of an execution in one call. This method serves as a higher-level convenience function that internally calls Start and Wait methods. It returns the result of the execution or an error if either starting or waiting fails, or if the context is canceled.

func (*Executor) Start 

func (e *Executor) Start(ctx context.Context, request *types.ExecutionRequest) error

Start begins the execution of a request by starting a Docker container.

func (*Executor) Wait 

func (e *Executor) Wait(
	ctx context.Context,
	executionID string,
) (<-chan *types.ExecutionResult, <-chan error)

Wait initiates a wait for the completion of a specific execution using its executionID. The function returns two channels: one for the result and another for any potential error. If the executionID is not found, an error is immediately sent to the error channel. Otherwise, an internal goroutine (doWait) is spawned to handle the asynchronous waiting. Callers should use the two returned channels to wait for the result of the execution or an error. This can be due to issues either beginning the wait or in getting the response. This approach allows the caller to synchronize Wait with calls to Start, waiting for the execution to complete.

func (*Executor) WaitForStatus 

func (e *Executor) WaitForStatus(
	ctx context.Context,
	executionID string,
	status types.ExecutionStatus,
	timeout *time.Duration,
) error

WaitForStatus waits for the execution to reach a specific status. It returns an error if the execution is not found or the status is unknown.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.