sbatch

Documentation

Overview

Package sbatch provides implementations of the SBatchService client.

Index

• type Bore
• type ContainerRun
• type EnvVar
• type ForRange
• type HTTPData
• type Job
• type JobResources
• type Module
• type ModuleInput
• type ModuleOutput
• type Mount
• type Mutation
• type NetworkInterface
• type Query
• type S3Data
• type Service
  • func NewService(client *http.Client, endpoint string) *Service
  • func (s *Service) Submit(ctx context.Context, job *Job) (string, error)
• type Step
• type StepAsyncLaunch
• type StepFor
• type StepRun
• type StepRunResources
• type StepUse
• type TransportData
• type VNet
• type VirtualNetwork
• type Wireguard
• type WireguardPeer

Types

type Bore

type Bore struct {
	// Bore server IP/Address.
	//
	// Deprecated: Use boreAddress.
	//
	// Go name: "Address".
	Address *string `json:"address,omitempty" yaml:"address" validate:"omitempty,ip|fqdn"`
	// Bore server IP/Address:Port.
	//
	// Go name: "BoreAddress".
	BoreAddress *string `json:"boreAddress,omitempty" yaml:"boreAddress" validate:"omitempty,hostname_port"`
	// The bore server port.
	//
	// Deprecated: Use boreAddress.
	//
	// Go name: "Port".
	Port *int `json:"port,omitempty" yaml:"port"`
	// Target port.
	//
	// Go name: "TargetPort".
	TargetPort int `json:"targetPort" yaml:"targetPort"`
	// Secret used to authenticate on a Bore server.
	//
	// This secret is used to identify the client.
	//
	// Go name: "Secret".
	Secret *string `json:"secret,omitempty" yaml:"secret"`
}

jkuri/bore tunnel Transport for StepRun.

Bore is a proxy to expose TCP sockets.

type ContainerRun

type ContainerRun struct {
	// Run the command inside a container with Enroot.
	//
	// Format: image:tag. Registry and authentication is not allowed on this field.
	//
	// If the default container runtime is used:
	//
	//   - Use an absolute path to load a squashfs file. By default, it will search inside $STORAGE_PATH. /input will be equivalent to $DEEPSQUARE_INPUT, /output is $DEEPSQUARE_OUTPUT
	//
	// If apptainer=true:
	//
	//   - Use an absolute path to load a sif file or a squashfs file. By default, it will search inside $STORAGE_PATH. /input will be equivalent to $DEEPSQUARE_INPUT, /output is $DEEPSQUARE_OUTPUT
	//
	// Examples:
	//
	//   - library/ubuntu:latest
	//   - /my.squashfs
	//
	// Go name: "Image".
	Image string `json:"image" yaml:"image" validate:"valid_container_image_url"`
	// [DEPRECATED] Mounts decribes a Bind Mount.
	//
	// Please use predefined mounts like $STORAGE_PATH, $DEEPSQUARE_TMP, ...
	//
	// Go name: "Mounts".
	Mounts []*Mount `json:"mounts,omitempty" yaml:"mounts,omitempty" validate:"omitempty,dive,required"`
	// Username of a basic authentication.
	//
	// Go name: "Username".
	Username *string `json:"username,omitempty" yaml:"username,omitempty"`
	// Password of a basic authentication.
	//
	// Go name: "Password".
	Password *string `json:"password,omitempty" yaml:"password,omitempty"`
	// Container registry host.
	//
	// Defaults to registry-1.docker.io.
	//
	// Go name: "Registry".
	Registry *string `json:"registry,omitempty" yaml:"registry,omitempty" validate:"omitempty,hostname"`
	// Run with Apptainer as Container runtime instead of Enroot.
	//
	// By running with apptainer, you get access Deepsquare-hosted images.
	//
	// When running Apptainer, the container file system is read-only.
	//
	// Defaults to false.
	//
	// Go name: "Apptainer".
	Apptainer *bool `json:"apptainer,omitempty" yaml:"apptainer,omitempty"`
	// Use DeepSquare-hosted images.
	//
	// By setting to true, apptainer will be set to true.
	//
	// Go name: "DeepsquareHosted".
	DeepsquareHosted *bool `json:"deepsquareHosted,omitempty" yaml:"deepsquareHosted,omitempty"`
	// X11 mounts /tmp/.X11-unix in the container.
	//
	// Go name: "X11".
	X11 *bool `json:"x11,omitempty" yaml:"x11,omitempty"`
	// Mount the home directories.
	//
	// Go name: "MountHome".
	MountHome *bool `json:"mountHome,omitempty" yaml:"mountHome,omitempty"`
	// Disable write permissions on the container root file system. Does not applies to mounts.
	//
	// Go name: "ReadOnlyRootFS"
	ReadOnlyRootFS *bool `json:"readOnlyRootFS,omitempty" yaml:"readOnlyRootFS,omitempty"`
}

type EnvVar

type EnvVar struct {
	// Key of the environment variable.
	//
	// Go name: "Key".
	Key string `json:"key" yaml:"key" validate:"required,valid_envvar_name,ne=PATH,ne=LD_LIBRARY_PATH"`
	// Value of the environment variable.
	//
	// Go name: "Value".
	Value string `json:"value" yaml:"value"`
}

An environment variable.

Accessible via: "$key". "Key" name must follows the POSIX specifications (alphanumeric with underscore).

type ForRange

type ForRange struct {
	// Begin is inclusive.
	//
	// Go name: "Begin".
	Begin int `json:"begin" yaml:"begin"`
	// End is inclusive.
	//
	// Go name: "End".
	End int `json:"end" yaml:"end"`
	// Increment counter by x count. If null, defaults to 1.
	//
	// Go name: "Increment".
	Increment *int `json:"increment,omitempty" yaml:"increment,omitempty"`
}

ForRange describes the parameter for a range loop.

type HTTPData

type HTTPData struct {
	// HTTP or HTTPS URL to a file.
	//
	// Go name: "URL".
	URL string `json:"url" yaml:"url" validate:"url"`
}

HTTPData describes the necessary variables to connect to a HTTP storage.

type Job

type Job struct {
	// Allocated resources for the job.
	//
	// Each resource is available as environment variables:
	// - $NTASKS: number of allowed parallel tasks
	// - $CPUS_PER_TASK: number of CPUs per task
	// - $MEM_PER_CPU: MB of memory per CPU
	// - $GPUS_PER_TASK: number of GPUs per task
	// - $GPUS: total number of GPUS
	// - $CPUS: total number of CPUS
	// - $MEM: total number of memory in MB
	//
	// Go name: "Resources".
	Resources *JobResources `json:"resources" yaml:"resources" validate:"required"`
	// Environment variables accessible for the entire job.
	//
	// Go name: "Env".
	Env []*EnvVar `json:"env,omitempty" yaml:"env,omitempty" validate:"omitempty,dive,required"`
	// EnableLogging enables the DeepSquare Grid Logger.
	//
	// Go name: "EnableLogging".
	EnableLogging *bool `json:"enableLogging,omitempty" yaml:"enableLogging,omitempty"`
	// Pull data at the start of the job.
	//
	// It is recommended to set the mode of the data by filling the `inputMode` field.
	//
	// Go name: "Input".
	Input *TransportData `json:"input,omitempty" yaml:"input,omitempty"`
	// InputMode takes an integer that will be used to change the mode recursively (chmod -R) of the input data.
	//
	// The number shouldn't be in octal but in decimal. A mode over 512 is not accepted.
	//
	// Common modes:
	//   - 511 (user:rwx group:rwx world:rwx)
	//   - 493 (user:rwx group:r-x world:r-x)
	//   - 448 (user:rwx group:--- world:---)
	//
	// If null, the mode won't change and will default to the source.
	//
	// Go name: "InputMode".
	InputMode *int `json:"inputMode,omitempty" yaml:"inputMode,omitempty" validate:"omitempty,lt=512"`
	// Group of steps that will be run sequentially.
	//
	// Go name: "Steps".
	Steps []*Step `json:"steps" yaml:"steps" validate:"dive,required"`
	// Push data at the end of the job.
	//
	// Continuous sync/push can be enabled using the `continuousOutputSync` flag.
	//
	// Go name: "Output".
	Output *TransportData `json:"output,omitempty" yaml:"output,omitempty"`
	// ContinuousOutputSync will push data during the whole job.
	//
	// This is useful when it is not desired to lose data when the job is suddenly stopped.
	//
	// ContinousOutputSync is not available with HTTP.
	//
	// Go name: "ContinuousOutputSync".
	ContinuousOutputSync *bool `json:"continuousOutputSync,omitempty" yaml:"continuousOutputSync,omitempty"`
	// A list of virtual network.
	//
	// Can only be used with network namespaces.
	//
	// Go name: "VirtualNetworks".
	VirtualNetworks []*VirtualNetwork `json:"virtualNetworks,omitempty" yaml:"virtualNetworks,omitempty" validate:"dive,required"`
}

A Job is a finite sequence of instructions.

type JobResources

type JobResources struct {
	// Number of tasks which are run in parallel.
	//
	// Can be greater or equal to 1.
	//
	// Go name: "Tasks".
	Tasks int `json:"tasks" yaml:"tasks" validate:"gte=1"`
	// Allocated CPUs per task.
	//
	// Can be greater or equal to 1.
	//
	// Go name: "CPUsPerTask".
	CPUsPerTask int `json:"cpusPerTask" yaml:"cpusPerTask" validate:"gte=1"`
	// Allocated memory (MB) per task.
	//
	// Can be greater or equal to 1.
	//
	// Go name: "MemPerCPU".
	MemPerCPU int `json:"memPerCpu" yaml:"memPerCpu" validate:"gte=1"`
	// Allocated GPUs for the whole job.
	//
	// Tasks can consume the GPUs by setting `GPUsPerTask` at step level.
	//
	// Can be greater or equal to 0.
	//
	// Go name: "GPUs".
	GPUs int `json:"gpus" yaml:"gpus" validate:"gte=0"`
}

JobResources are the allocated resources for a job in a cluster.

type Module

type Module struct {
	// Name of the module.
	//
	// Go name: "Name".
	Name string `json:"name" yaml:"name"`
	// Description of the module.
	//
	// Go name: "Description".
	Description string `json:"description" yaml:"description"`
	// Minimum job resources.
	//
	// Go name: "MinimumResources".
	MinimumResources *JobResources `json:"minimumResources" yaml:"minimumResources"`
	// List of allowed arguments.
	//
	// Go name: "Inputs".
	Inputs []*ModuleInput `json:"inputs,omitempty" yaml:"inputs,omitempty" validate:"omitempty,dive,required"`
	// List of exported environment variables.
	//
	// Go name: "Outputs".
	Outputs []*ModuleOutput `json:"outputs,omitempty" yaml:"outputs,omitempty" validate:"omitempty,dive,required"`
	// Steps of the module.
	//
	// Go name: "Steps".
	Steps []*Step `json:"steps" yaml:"steps" validate:"dive,required"`
}

A module is basically a group of steps.

The module.yaml file goes through a templating engine first before getting parsed. So some variables are available:

- `{{ .Job }}` and its childs, which represent the Job object using the module. Can be useful if you want to dynamically set an value based on the job. - `{{ .Step }}` and its childs, which represent the Step object using the module. Can be useful if you want the step name.

If you want your user to pass custom steps, you can use `{{- .Step.Use.Steps | toYaml | nindent <n> }}` which is the group of steps.

Example:

```yaml # module.yaml steps:

• name: my step {{- .Step.Use.Steps | toYaml | nindent 2 }}
• name: my other step

```

```yaml # job.yaml steps:

• name: module use: source: git/my-module steps:
• name: step by user
• name: another step by user

```

Will render:

```yaml # module.yaml steps:

• name: my step
• name: step by user
• name: another step by user
• name: my other step

```

Notice that the templating follows the Go format. You can also apply [sprig](http://masterminds.github.io/sprig/) templating functions.

To outputs environment variables, just append KEY=value to the "${DEEPSQUARE_ENV}" file, like this:

``` echo "KEY=value" >> "${DEEPSQUARE_ENV}" ```

type ModuleInput

type ModuleInput struct {
	// Name of the input.
	//
	// Go name: "Key".
	Key string `json:"key" yaml:"key" validate:"valid_envvar_name,ne=PATH,ne=LD_LIBRARY_PATH"`
	// Description of the input.
	//
	// Go name: "Description".
	Description string `json:"description" yaml:"description"`
	// Default value.
	//
	// If not set, will default to empty string.
	//
	// Go name: "Default".
	Default *string `json:"default,omitempty" yaml:"default,omitempty"`
}

type ModuleOutput

type ModuleOutput struct {
	// Name of the output.
	//
	// Go name: "Key".
	Key string `json:"key" yaml:"key" validate:"valid_envvar_name,ne=PATH,ne=LD_LIBRARY_PATH"`
	// Description of the output.
	//
	// Go name: "Description".
	Description string `json:"description" yaml:"description"`
}

type Mount

type Mount struct {
	// Directory on the host to be mounted inside the container.
	//
	// Go name: "HostDir".
	HostDir string `json:"hostDir" yaml:"hostDir" validate:"startswith=/"`
	// Target directory inside the container.
	//
	// Go name: "ContainerDir".
	ContainerDir string `json:"containerDir" yaml:"containerDir" validate:"startswith=/"`
	// Options modifies the mount options.
	//
	// Accepted: ro, rw
	//
	// Go name: "Options".
	Options string `json:"options" yaml:"options" validate:"omitempty,oneof=rw ro"`
}

DEPRECATED: Mount decribes a Bind Mount.

Mount is now deprecated. Please use predefined mounts like $STORAGE_PATH, $DEEPSQUARE_TMP, ...

type Mutation

type Mutation struct {
}

type NetworkInterface

type NetworkInterface struct {
	// Use the wireguard transport.
	//
	// Go name: "Wireguard".
	Wireguard *Wireguard `json:"wireguard,omitempty" yaml:"wireguard,omitempty"`
	// Use the bore transport.
	//
	// Go name: "Bore".
	Bore *Bore `json:"bore,omitempty" yaml:"bore,omitempty"`
	// Use a DeepSquare-managed virtual network for inter-step communication.
	//
	// It uses Wireguard to interconnect the steps. The communication are encrypted.
	//
	// Go name: "VNet".
	VNet *VNet `json:"vnet,omitempty" yaml:"vnet,omitempty"`
}

Connect a network interface on a StepRun.

The network interface is connected via slirp4netns.

If using wireguard, please mapUid to root (mapUid=0).

type Query

type Query struct {
}

type S3Data

type S3Data struct {
	// S3 region. Example: "us‑east‑2".
	//
	// Go name: "Region".
	Region string `json:"region" yaml:"region"`
	// The S3 Bucket URL. Must not end with "/".
	//
	// Example: "s3://my-bucket".
	//
	// Go name: "BucketURL".
	BucketURL string `json:"bucketUrl" yaml:"bucketUrl" validate:"url,startswith=s3://,endsnotwith=/"`
	// The absolute path to a directory/file inside the bucket. Must start with "/".
	//
	// Go name: "Path".
	Path string `json:"path" yaml:"path" validate:"startswith=/"`
	// An access key ID for the S3 endpoint.
	//
	// Go name: "AccessKeyID".
	AccessKeyID string `json:"accessKeyId" yaml:"accessKeyId"`
	// A secret access key for the S3 endpoint.
	//
	// Go name: "SecretAccessKey".
	SecretAccessKey string `json:"secretAccessKey" yaml:"secretAccessKey"`
	// A S3 Endpoint URL used for authentication. Example: https://s3.us‑east‑2.amazonaws.com
	//
	// Go name: "EndpointURL".
	EndpointURL string `json:"endpointUrl" yaml:"endpointUrl" validate:"url"`
	// DeleteSync removes destination files that doesn't correspond to the source.
	//
	// This applies to any type of source to any type of destination (s3 or filesystem).
	//
	// See: s5cmd sync --delete.
	//
	// If null, defaults to false.
	//
	// Go name: "DeleteSync".
	DeleteSync *bool `json:"deleteSync,omitempty" yaml:"deleteSync,omitempty"`
}

S3Data describes the necessary variables to connect to a S3 storage.

type Service

type Service struct {
	*http.Client
	// contains filtered or unexported fields
}

Service is a client for the SBatchService.

func NewService

func NewService(client *http.Client, endpoint string) *Service

NewService creates a new Service.

func (*Service) Submit

func (s *Service) Submit(ctx context.Context, job *Job) (string, error)

Submit submits a job to the SBatchService.

type Step

type Step struct {
	// Name of the instruction.
	//
	// Is used for debugging.
	//
	// Go name: "Name".
	Name *string `json:"name,omitempty" yaml:"name,omitempty"`
	// Depends on wait for async tasks to end before launching this step.
	//
	// DependsOn uses the `handleName` property of a `StepAsyncLaunch`.
	//
	// Only steps at the same level can be awaited.
	//
	// BE WARNED: Uncontrolled `dependsOn` may results in dead locks.
	//
	// Go name: "DependsOn".
	DependsOn []string `json:"dependsOn,omitempty" yaml:"dependsOn,omitempty" validate:"omitempty,dive,alphanum_underscore"`
	// "If" is a boolean test that skips the step if the test is false.
	//
	// The test format is bash and variables such as $PATH or $(pwd) can be expanded.
	//
	// Note that "If" will be run after the "DependsOn".
	//
	// Example: '3 -eq 3 && "${TEST}" = "test"'.
	//
	// Go name: "If".
	If *string `json:"if,omitempty" yaml:"if,omitempty"`
	// Group of steps that will be run sequentially.
	//
	// Is exclusive with "for", "launch", "use", "run".
	//
	// Go name: "Steps".
	Steps []*Step `json:"steps,omitempty" yaml:"steps,omitempty" validate:"omitempty,dive,required"`
	// Run a command if not null.
	//
	// Is exclusive with "for", "launch", "use", "steps".
	//
	// Go name: "Run".
	Run *StepRun `json:"run,omitempty" yaml:"run,omitempty"`
	// Run a for loop if not null.
	//
	// Is exclusive with "run", "launch", "use", "steps".
	//
	// Go name: "For".
	For *StepFor `json:"for,omitempty" yaml:"for,omitempty"`
	// Launch a background process to run a group of commands if not null.
	//
	// Is exclusive with "run", "for", "use", "steps".
	//
	// Go name: "Launch".
	Launch *StepAsyncLaunch `json:"launch,omitempty" yaml:"launch,omitempty"`
	// Use a third-party group of steps.
	//
	// Is exclusive with "run", "for", "launch", "steps".
	//
	// Go name: "Use".
	Use *StepUse `json:"use,omitempty" yaml:"use,omitempty"`
	// Group of steps that will be run sequentially on error.
	//
	// Go name: "Catch".
	Catch []*Step `json:"catch,omitempty" yaml:"catch,omitempty" validate:"omitempty,dive,required"`
	// Group of steps that will be run sequentially after the group of steps or command finishes.
	//
	// Go name: "Finally".
	Finally []*Step `json:"finally,omitempty" yaml:"finally,omitempty" validate:"omitempty,dive,required"`
}

Step is one instruction.

type StepAsyncLaunch

type StepAsyncLaunch struct {
	// HandleName is the name used to await (dependsOn field of the Step).
	//
	// Naming style is snake_case. Case is insensitive. No symbol allowed.
	//
	// Go name: "HandleName".
	HandleName *string `json:"handleName,omitempty" yaml:"handleName,omitempty" validate:"omitempty,alphanum_underscore"`
	// SignalOnParentStepExit sends a signal to the step and sub-steps when the parent step ends.
	//
	// This function can be used as a cleanup function to avoid a zombie process.
	//
	// Zombie processes will continue to run after the main process dies and therefore will not stop the job.
	//
	// If null, SIGTERM will be sent. If 0, no signal will be sent.
	//
	// Current signal :
	//
	// 1 SIGHUP Hang-up detected on the control terminal or death of the control process.
	// 2 SIGINT Abort from keyboard
	// 3 SIGQUIT Quit the keyboard
	// 9 SIGKILL If a process receives this signal, it must quit immediately and will not perform any cleaning operations.
	// 15 SIGTERM Software stop signal
	//
	// It is STRONGLY RECOMMENDED to use SIGTERM to gracefully exit a process. SIGKILL is the most abrupt and will certainly work.
	//
	// If no signal is sent, the asynchronous step will be considered a fire and forget asynchronous step and will have to terminate itself to stop the job.
	//
	// WARNING: the "no signal sent" option is subject to removal to avoid undefined behavior. Please refrain from using it.
	//
	// Go name: "SignalOnParentStepExit".
	SignalOnParentStepExit *int `json:"signalOnParentStepExit,omitempty" yaml:"signalOnParentStepExit,omitempty"`
	// Steps are run sequentially.
	//
	// Go name: "Steps".
	Steps []*Step `json:"steps" yaml:"steps" validate:"dive,required"`
}

StepAsyncLaunch describes launching a background process.

StepAsyncLaunch will be awaited at the end of the job.

type StepFor

type StepFor struct {
	// Do a parallel for loop. Each iteration is run in parallel.
	//
	// Go name: "Parallel".
	Parallel bool `json:"parallel" yaml:"parallel"`
	// Item accessible via the {{ .Item }} variable. Index accessible via the $item variable.
	//
	// Exclusive with "range".
	//
	// Go name: "Items".
	Items []string `json:"items,omitempty" yaml:"items,omitempty"`
	// Index accessible via the $index variable.
	//
	// Exclusive with "items".
	//
	// Go name: "Range".
	Range *ForRange `json:"range,omitempty" yaml:"range,omitempty"`
	// Steps are run sequentially in one iteration.
	//
	// Go name: "Steps".
	Steps []*Step `json:"steps" yaml:"steps" validate:"dive,required"`
}

StepFor describes a for loop.

type StepRun

type StepRun struct {
	// Command specifies a shell script or CMD.
	//
	// If container is used, `command` and `shell` automatically overwrite the ENTRYPOINT and CMD.
	//
	// If you want to execute the default ENTRYPOINT and CMD, set the `command` empty.
	//
	// If you want to execute the default ENTRYPOINT with a custom CMD, set the `command` to your args and set the `shell` to `ENTRYPOINT`.
	//
	// Right now, ENTRYPOINT cannot be overriden for security reasons.
	//
	// Go name: "Command".
	Command string `json:"command" yaml:"command"`
	// Shell to use.
	//
	// Use "ENTRYPOINT" to use the default ENTRYPOINT.
	//
	// Accepted: /bin/bash, /bin/ash, /bin/sh, ENTRYPOINT
	// Default: /bin/sh
	//
	// Go name: "Shell".
	Shell *string `json:"shell,omitempty" yaml:"shell,omitempty" validate:"omitempty,oneof=/bin/bash /bin/ash /bin/sh ENTRYPOINT"`
	// Allocated resources for the command.
	//
	// Go name: "Resources".
	Resources *StepRunResources `json:"resources,omitempty" yaml:"resources,omitempty"`
	// Container definition.
	//
	// If null, run on the host.
	//
	// Go name: "Container".
	Container *ContainerRun `json:"container,omitempty" yaml:"container,omitempty"`
	// Type of core networking functionality.
	//
	// Either: "host" (default) or "slirp4netns" (rootless network namespace) or "pasta" (simple rootless network namespace)
	//
	// "slirp4netns" uses "slirp" to forward traffic from a network namespace to the host.
	//
	// "pasta" is an alternative to "slirp4netns" and uses "passt" to forward traffic from a network namespace to the host.
	//
	// Go name: "Network".
	Network *string `json:"network,omitempty" yaml:"network,omitempty" validate:"omitempty,oneof=host slirp4netns pasta"`
	// Configuration for the DNS in "slirp4netns" or "pasta" mode.
	//
	// ONLY enabled if network is "slirp4netns" or "pasta".
	//
	// A comma-separated list of DNS IP.
	//
	// Go name: "DNS".
	DNS []string `json:"dns,omitempty" yaml:"dns,omitempty" validate:"omitempty,dive,ip"`
	// Add custom network interfaces.
	//
	// ONLY enabled if network is "slirp4netns" or "pasta".
	//
	// You may need to map to root to be able to create network interfaces like Wireguard.
	//
	// The default network interface is net0, which is a TAP interface connecting the host and the network namespace.
	//
	// Go name: "CustomNetworkInterfaces".
	CustomNetworkInterfaces []*NetworkInterface `json:"customNetworkInterfaces,omitempty" yaml:"customNetworkInterfaces,omitempty" validate:"omitempty,dive,required"`
	// Environment variables accessible over the command.
	//
	// Go name: "Env".
	Env []*EnvVar `json:"env,omitempty" yaml:"env,omitempty" validate:"omitempty,dive,required"`
	// Remap UID. Does not grant elevated system permissions, despite appearances.
	//
	// MapUID doesn't work very well with Apptainer. You can still map to root, but you cannot map to an unknown user.
	//
	// Go name: "MapUID".
	MapUID *int `json:"mapUid,omitempty" yaml:"mapUid,omitempty"`
	// Remap GID. Does not grant elevated system permissions, despite appearances.
	//
	// Go name: "MapGID".
	MapGID *int `json:"mapGid,omitempty" yaml:"mapGid,omitempty"`
	// Working directory.
	//
	// If the "default" (Enroot) container runtime is used, it will use the `--container-workdir` flag.
	//
	// If the "apptainer" container runtime is used, the `--pwd` flag will be passed.
	//
	// If no container runtime is used, `cd` will be executed first.
	//
	// If null, default to use $STORAGE_PATH as working directory.
	//
	// Go name: "WorkDir".
	WorkDir *string `json:"workDir,omitempty" yaml:"workDir,omitempty" validate:"omitempty,startswith=/"`
	// DisableCPUBinding disables process affinity binding to tasks.
	//
	// Can be useful when running MPI jobs.
	//
	// If null, defaults to false.
	//
	// Go name: "DisableCPUBinding".
	DisableCPUBinding *bool `json:"disableCpuBinding,omitempty" yaml:"disableCpuBinding,omitempty"`
	// MPI selection.
	//
	// Must be one of: none, pmix_v4, pmi2.
	//
	// If null, will default to infrastructure provider settings (which may not be what you want).
	//
	// Go name: "Mpi".
	Mpi *string `json:"mpi,omitempty" yaml:"mpi,omitempty" validate:"omitempty,oneof=none pmix_v4 pmi2"`
}

StepRun is one script executed with the shell.

A temporary shared storage is accessible through the $STORAGE_PATH environment variable.

Availables caches can be used by invoking one of the following environment variable:

| Environment variables | Lifecycle | | --------------------------------------- | -------------------------------- | | STORAGE_PATH | job duration | | DEEPSQUARE_TMP or DEEPSQUARE_SHARED_TMP | provider's policy | | DEEPSQUARE_SHARED_WORLD_TMP | provider's policy | | DEEPSQUARE_DISK_TMP | node reboot or provider's policy | | DEEPSQUARE_DISK_WORLD_TMP | node reboot or provider's policy |

echo "KEY=value" >> "$DEEPSQUARE_ENV" can be used to share environment variables between steps.

$DEEPSQUARE_INPUT is the path that contains imported files.

$DEEPSQUARE_OUTPUT is the staging directory for uploading files.

type StepRunResources

type StepRunResources struct {
	// Number of tasks which are run in parallel.
	//
	// Can be greater or equal to 1.
	//
	// If null, default to 1.
	//
	// Go name: "Tasks".
	Tasks *int `json:"tasks,omitempty" yaml:"tasks,omitempty" validate:"omitempty,gte=1"`
	// Allocated CPUs per task.
	//
	// Can be greater or equal to 1.
	//
	// If null, defaults to the job resources.
	//
	// Go name: "CPUsPerTask".
	CPUsPerTask *int `json:"cpusPerTask,omitempty" yaml:"cpusPerTask,omitempty" validate:"omitempty,gte=1"`
	// Allocated memory (MB) per task.
	//
	// Can be greater or equal to 1.
	//
	// If null, defaults to the job resources.
	//
	// Go name: "MemPerCPU".
	MemPerCPU *int `json:"memPerCpu,omitempty" yaml:"memPerCpu,omitempty" validate:"omitempty,gte=1"`
	// Allocated GPUs per task.
	//
	// Can be greater or equal to 0.
	//
	// If null, defaults to 0.
	//
	// Go name: "GPUsPerTask".
	GPUsPerTask *int `json:"gpusPerTask,omitempty" yaml:"gpusPerTask,omitempty" validate:"omitempty,gte=0"`
}

StepRunResources are the allocated resources for a command in a job.

type StepUse

type StepUse struct {
	// Source of the group of steps.
	//
	// Syntax: <url>@<tag/hash>
	//
	// Example: github.com/example/my-module@v1
	// Example: github.com/example/module-monorepo/my-module@v1
	//
	// The host must be a git repository accessible via HTTPS.
	// The path must indicates a directory. For example, `/my-module` indicates the root directory of the repository `my-module`.
	// `module-monorepo/my-module` indicates the subdirectory `my-module` of the repository `module-monorepo`.
	//
	// Go name: "Source".
	Source string `json:"source" yaml:"source"`
	// Arguments to be passed as inputs to the group of steps.
	//
	// Go name: "Args".
	Args []*EnvVar `json:"args,omitempty" yaml:"args,omitempty"`
	// Environment variables exported with be prefixed with the value of this field.
	//
	// Exemple: If exportEnvAs=MY_MODULE, and KEY is exported. Then you can invoke ${MY_MODULE_KEY} environment variable.
	//
	// Go name: "ExportEnvAs".
	ExportEnvAs *string `json:"exportEnvAs,omitempty" yaml:"exportEnvAs,omitempty" validate:"omitempty,valid_envvar_name,ne=PATH,ne=LD_LIBRARY_PATH"`
	// Additional children steps to the module.
	//
	// If the module allow children steps, these steps will be passed to the module to replace {{ .Step.Run.Steps }}.
	//
	// Go name: "Steps".
	Steps []*Step `json:"steps,omitempty" yaml:"steps,omitempty" validate:"omitempty,dive,required"`
}

type TransportData

type TransportData struct {
	// Use http to download a file or archive, which will be autoextracted.
	//
	// Go name: "HTTP".
	HTTP *HTTPData `json:"http,omitempty" yaml:"http,omitempty"`
	// Use s3 to sync a file or directory.
	//
	// Go name: "S3".
	S3 *S3Data `json:"s3,omitempty" yaml:"s3,omitempty"`
}

type VNet

type VNet struct {
	// Name of the network to be used. Must exists.
	//
	// See Job.Networks.
	//
	// Go name: "Name".
	Name string `json:"name" yaml:"name,omitempty"`
	// Address (CIDR) of the interface.
	//
	// Example: "10.0.0.2/24" which means
	//
	//   - The interface's IP is 10.0.0.2.
	//
	//   - Route packets with destination 10.0.0.0/24 to that interface.
	//
	// Go name: "Address"
	Address string `json:"address" yaml:"address,omitempty" validate:"cidr"`
}

Use VNet as network interface.

type VirtualNetwork

type VirtualNetwork struct {
	// Name of the virtual network.
	//
	// Use this name to reference the network.
	//
	// Go name: "Name".
	Name string `json:"name" yaml:"name,omitempty"`
	// Gateway address (CIDR). Note this does not forward to the internet. This is only used for NAT traversal.
	//
	// Example: "10.0.0.1/24". IPv6 is also supported.
	//
	// Go name: "GatewayAddress".
	GatewayAddress string `json:"gatewayAddress" yaml:"gatewayAddress,omitempty" validate:"cidr"`
}

A virtual network is a network that can be used to connect network namespaces.

For now, the virtual network use

type Wireguard

type Wireguard struct {
	// The IP addresses of the wireguard interface.
	//
	// Format is a CIDRv4 (X.X.X.X/X) or CIDRv6.
	//
	// Recommendation is to take one IP from the 10.0.0.0/24 range (example: 10.0.0.2/24).
	//
	// Go name: "Address".
	Address []string `json:"address,omitempty" yaml:"address,omitempty" validate:"omitempty,dive,cidr"`
	// The client private key.
	//
	// Go name: "PrivateKey".
	PrivateKey string `json:"privateKey" yaml:"privateKey"`
	// The peers connected to the wireguard interface.
	//
	// Go name: "Peers".
	Peers []*WireguardPeer `json:"peers,omitempty" yaml:"peers,omitempty" validate:"omitempty,dive,required"`
}

Wireguard VPN Transport for StepRun.

The Wireguard VPN can be used as a gateway for the steps. All that is needed is a Wireguard server outside the cluster that acts as a public gateway.

The interfaces are named wg0, wg1, ..., wgN.

Wireguard transport uses UDP hole punching to connect to the VPN Server.

Disabled settings: PreUp, PostUp, PreDown, PostDown, ListenPort, Table, MTU, SaveConfig.

If these features are necessary, please do contact DeepSquare developpers!

type WireguardPeer

type WireguardPeer struct {
	// The peer private key.
	//
	// Go name: "PublicKey".
	PublicKey string `json:"publicKey" yaml:"publicKey"`
	// The peer pre-shared key.
	//
	// Go name: "PreSharedKey".
	PreSharedKey *string `json:"preSharedKey,omitempty" yaml:"preSharedKey,omitempty"`
	// Configuration of wireguard routes.
	//
	// Format is a CIDRv4 (X.X.X.X/X) or CIDRv6.
	//
	// 0.0.0.0/0 (or ::/0) would forward all packets to the tunnel. If you plan to use the Wireguard VPN as a gateway, you MUST set this IP range.
	//
	// <server internal IP>/32 (not the server's public IP) would forward all packets to the tunnel with the server IP as the destination. MUST be set.
	//
	// <VPN IP range> would forward all packets to the tunnel with the local network as the destination. Useful if you want peers to communicate with each other and want the gateway to act as a router.
	//
	// Go name: "AllowedIPs".
	AllowedIPs []string `json:"allowedIPs,omitempty" yaml:"allowedIPs,omitempty" validate:"omitempty,dive,cidr"`
	// The peer endpoint.
	//
	// Format is IP:port.
	//
	// This would be the Wireguard server.
	//
	// Go name: "Endpoint".
	Endpoint *string `json:"endpoint,omitempty" yaml:"endpoint,omitempty" validate:"omitempty,hostname_port"`
	// Initiate the handshake and re-initiate regularly.
	//
	// Takes seconds as parameter. 25 seconds is recommended.
	//
	// You MUST set the persistent keepalive to enables UDP hole-punching.
	//
	// Go name: "PersistentKeepalive".
	PersistentKeepalive *int `json:"persistentKeepalive,omitempty" yaml:"persistentKeepalive,omitempty"`
}

A Wireguard Peer.