dockertracker

package
v0.3.36 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 19, 2024 License: Apache-2.0 Imports: 25 Imported by: 2

README

Docker Tracker

Introduction

Docker Tracker implements the JobTracker interface from Go drmaa2os. It allows to use Docker as a backend for managing jobs as containers from the DRMAA2 interface. It can also be used directly. The package also contains an implementation of the Monitorer interface so that it can be used in a DRMAA2 monitoring session.

Functionality

Docker Tracker is an API that enables the use of Docker as a backend for managing jobs as containers through the DRMAA2 interface. It provides an implementation of the JobTracker interface from Go drmaa2os, allowing easy job control and management within Docker.

The functionality of Docker Tracker includes:

  1. Starting Docker containers using the DRMAA2 JobTemplate which requires a JobCategory (corresponding to a Docker image) and a RemoteCommand (the command to be executed within the Docker image).

  2. Providing job control functions such as suspend, resume, and terminate for managing Docker containers.

  3. Mapping DRMAA2 Job Control commands to corresponding Docker commands for seamless integration.

  4. Mapping DRMAA2 State to Docker State to provide a consistent view of the container's status.

  5. Allowing the removal of installed containers through the DeleteJob command, which is equivalent to docker rm.

  6. Supporting Job Template Mapping to efficiently map between the JobTemplate and the Docker container configuration request.

  7. Implementing Job Array functionality by creating multiple tasks sequentially in a loop, since Docker does not support Array Jobs natively.

Please note that Docker Tracker does not pull container images automatically, and the required images must be pulled before using the tool. Additionally, some DRMAA2 functionalities, such as Hold and Release, are not supported in Docker Tracker due to limitations in Docker.

For the case a Docker image needs to be pulled programmatically the OS process backend can be used.

Basic Usage

A JobTemplate requires:

* JobCategory -> which maps to an installed Docker image
* RemoteCommand -> which is the command executed within the given Docker image
Job Control Mapping
DRMAA2 Job Control Docker
Suspend Signal: SIGSTOP
Resume Signal: SIGCONT
Terminate Signal: SIGKILL
Hold Unsupported
Release Unsupported
State Mapping
DRMAA2 State Docker State
Failed OOMKilled
Failed or Done depending on exit code Exited
Failed or Done depending on exit code Dead
Suspended Paused
Running Running
Queued Restarting
Undetermined other

DeleteJob

DeleteJob equals docker rm and is removing an installed container. It must be terminated / finished before.

Job Template Mapping

Mapping between the job template and the Docker container config request:

DRMAA2 JobTemplate Docker Container Config Request
RemoteCommand Cmd[0]
Args Cmd[1:]
JobCategory Image
CandidateMachines[0] Hostname
WorkingDir WorkingDir
JobEnvironment (k: v) Env ("k=v")
StageInFiles key: hostPath -v hostPath:containerPath
ErrorPath Writes stderr into a local file (not a file in the container).
OutputPath Writes stdout into a local file (not a file in the container).
Extension: "user" User / must exist in container if set
Extension: "exposedPorts" -p / multiple entries are splitted with ","
Extension: "net" --net / like "host"
Extension: "privileged" --privileged / "true" when enabled, default "false"
Extension: "restart" --restart / like "unless-stopped", default "no" / use with care
Extension: "ipc" Value like "host": --ipc "host"
Extension: "uts" --uts "host"
Extension: "pid" --pid "host"
Extension: "rm" --rm "true" or "TRUE"
Extension: "gpus" Sets GPU device request for nvidia driver. Value must be "all" or a number
Extension: "shm-size" Human-readable string representing an amount of RAM in bytes, kibibytes, mebibytes, gibibytes, or tebibytes
Extension: "ulimit" Comma separted list of ulimits: "memlock=-1,stack=67108864"

Check convert.go for more details. Add your own extention and create a pull request.

If more extensions needed just open an issue.

Note that the image must be available (pulled already)!

Job Info Mapping
DRMAA2 JobInfo Docker Container Information
ID Container ID
Slots 1 (fixed value)
AllocatedMachines Config.Hostname
ExitStatus State.ExitCode
FinishTime State.FinishedAt
DispatchTime State.StartedAt
State Mapped from Container State
SubmissionTime Container Creation Time
JobOwner Config.User
ExtensionList (workingdir) Config.WorkingDir
ExtensionList (commandline) Config.Cmd (joined as a string)
ExtensionList (category) Config.Image
Job Arrays

Since Array Jobs are not supported by Docker the job array functionality is implemented by creating n tasks sequentially in a loop. The array job ID contains all IDs of the created Docker containers.

Documentation

Index

Constants

View Source 
const (
	ContainerLabelJobTemplate = "drmaa2jobtemplate"
)

Variables

This section is empty.

Functions

func NewAllocator added in v0.3.0 

func NewAllocator() *allocator

func ReadJobTemplateFromLabel added in v0.3.26 

func ReadJobTemplateFromLabel(containerID string) (drmaa2interface.JobTemplate, error)

ReadJobTemplateFromLabel reads the "drmaa2jobtemplate" label from the specified container. Then it decodes the base64/json encoded JobTemplate and returns it. Fo encoding see jobTemplateToContainerConfig().

Types

type DockerTracker 

type DockerTracker struct {
	// contains filtered or unexported fields
}

func New 

func New(jobsession string) (*DockerTracker, error)

New creates a new DockerTracker. How the Docker client is configured can be influenced by (from the Docker Documentation (https://github.com/moby/moby/blob/master/client/client.go)): "Use DOCKER_HOST to set the url to the docker server. 

Use DOCKER_API_VERSION to set the version of the API to reach, leave empty for latest.
Use DOCKER_CERT_PATH to load the TLS certificates from.
Use DOCKER_TLS_VERIFY to enable or disable TLS verification, off by default."

func (*DockerTracker) AddArrayJob 

func (dt *DockerTracker) AddArrayJob(jt drmaa2interface.JobTemplate, begin int, end int, step int, maxParallel int) (string, error)

func (*DockerTracker) AddJob 

func (dt *DockerTracker) AddJob(jt drmaa2interface.JobTemplate) (string, error)

func (*DockerTracker) CloseMonitoringSession added in v0.3.16 

func (dt *DockerTracker) CloseMonitoringSession(name string) error

func (*DockerTracker) DeleteJob 

func (dt *DockerTracker) DeleteJob(jobid string) error

DeleteJob removes a container so it is no longer in docker ps -a (and therefore not in the job list).

func (*DockerTracker) GetAllJobIDs added in v0.3.16 

func (dt *DockerTracker) GetAllJobIDs(filter *drmaa2interface.JobInfo) ([]string, error)

func (*DockerTracker) GetAllMachines added in v0.3.16 

func (dt *DockerTracker) GetAllMachines(filter []string) ([]drmaa2interface.Machine, error)

func (*DockerTracker) GetAllQueueNames added in v0.3.16 

func (dt *DockerTracker) GetAllQueueNames(filter []string) ([]string, error)

func (*DockerTracker) JobControl 

func (dt *DockerTracker) JobControl(jobid, state string) error

func (*DockerTracker) JobInfo 

func (dt *DockerTracker) JobInfo(jobid string) (ji drmaa2interface.JobInfo, err error)

func (*DockerTracker) JobInfoFromMonitor added in v0.3.16 

func (dt *DockerTracker) JobInfoFromMonitor(id string) (ji drmaa2interface.JobInfo, err error)

JobInfoFromMonitor might collect job state and job info in a different way as a JobSession with persistent storage does

func (*DockerTracker) JobState 

func (dt *DockerTracker) JobState(jobid string) (drmaa2interface.JobState, string, error)

func (*DockerTracker) JobTemplate added in v0.3.26 

func (dt *DockerTracker) JobTemplate(jobID string) (drmaa2interface.JobTemplate, error)

JobTemplate returns the JobTemplate for the given jobID. This implements the JobTemplater interface for the DockerTracker.

func (*DockerTracker) ListArrayJobs 

func (dt *DockerTracker) ListArrayJobs(id string) ([]string, error)

func (*DockerTracker) ListJobCategories 

func (dt *DockerTracker) ListJobCategories() ([]string, error)

ListJobCategories lists all container images available to run commands on.

func (*DockerTracker) ListJobs 

func (dt *DockerTracker) ListJobs() ([]string, error)

func (*DockerTracker) OpenMonitoringSession added in v0.3.16 

func (dt *DockerTracker) OpenMonitoringSession(name string) error

func (*DockerTracker) Wait 

func (dt *DockerTracker) Wait(jobid string, timeout time.Duration, states ...drmaa2interface.JobState) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.