kueueleuleu

README

kueueleuleu

Run containers sequentially inside Kubernetes Pods, Jobs, CronJobs.

The name kueueleuleu (\kø lø.lø\) comes from the French queue leu-leu, meaning "one after the other" or "in a single file". The initial k is obviously for kubernetes.

Motivation

As of now, Kubernetes does not support running containers inside a Pod sequentially, despite this being a common thing asked by users:

• How to run containers sequentially as a Kubernetes job?
• How to run multiple Kubernetes jobs in sequence?
• Kubernetes — Sequencing Container Startup, by Paul Dally
• How to start one container before staring another in the same pod?

Without an external way to control the containers run order, all containers inside a Pod start and run at the same time.

Existing known solutions

TL;DR: Convert your containers to init containers OR use external workflow solutions.

Use init containers

Init containers always run sequentially:

If you specify multiple init containers for a Pod, kubelet runs each init container sequentially. Each init container must succeed before the next can run.

Thus, a common workaround to allow running containers sequentially is to convert all containers to init containers, but:

• this requires to change the pod spec ourselves, which is not always possible or might feel "hacky"
• init containers have some restrictions (e.g. does not support probes, handles resource requests/limits differently than regular containers)

Use external tools designed for workflows

There are some solutions to define complex containers workflows (and among them, running containers sequentially):

• Argo Workflows
• Tekton

But, for such a simple use-case (run containers one after the other), they might seem overkill:

• need to install (and maintain) a fully-fledged tool
• need to use specific CRDs (Argo Workflow or Tekton Pipeline)

Kueueleuleu!

Unlike the known solutions mentioned above, this simple library aims to:

• hide the sequential containers orchestration logic from the user
• let users manipulate raw kubernetes objects (Pods, Jobs, CronJobs), so they can use everything natively supported by Kubernetes (volumes, runtime classes, etc.)
• provide an easy way to convert any of these kubernetes objects to kueueleuleu objects

How to use?

This library requires Go >= 1.21.

Using the CLI

go install github.com/norbjd/kueueleuleu/cmd/kueueleuleu@main

# create a simple pod with two containers: by default, both will run at the same time
cat > simplepod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: two-steps-pod
spec:
  containers:
    - name: step1
      image: alpine
      command:
        - sh
        - -c
      args:
        - echo "start step1" && sleep 5 && echo "end step1"
    - name: step2
      image: alpine
      command:
        - sh
        - -c
      args:
        - echo "start step2" && sleep 2 && echo "end step2"
  restartPolicy: Never
EOF

# convert this pod to run containers sequentially and apply
# if you like chaining commands, this is similar to: cat simplepod.yaml | ./kueueleuleu -f - | kubectl apply -f -
# if kueueleuleu is not in your PATH, replace with $GOPATH/bin/kueueleuleu (default when running go install)
kueueleuleu -f simplepod.yaml | kubectl apply -f -

Once the pod is finished, check the logs (kubectl logs two-steps-pod --all-containers --timestamps | sort) to see containers have been executed sequentially (first, step1, and then step2):

2023-12-29T13:32:50.488543841Z 2023/12/29 13:32:50 Entrypoint initialization
2023-12-29T13:32:52.139027305Z start step1
2023-12-29T13:32:57.139823758Z end step1
2023-12-29T13:32:57.392700813Z start step2
2023-12-29T13:32:59.393583491Z end step2

[!NOTE] The first log is related to internal logic: the entrypoint mentioned is here to order the containers' execution. You can ignore this, unless you are interested in the internals (see "Internals" section).

Without kueueleuleu (kubectl apply -f simplepod.yaml), containers logs are intertwined because both containers are running at the same time:

2023-12-29T13:33:43.942542903Z start step1
2023-12-29T13:33:44.848147388Z start step2
2023-12-29T13:33:46.848812173Z end step2
2023-12-29T13:33:48.943165982Z end step1

Conversion also work with Jobs and CronJobs, and even with YAML files containing multiple resources (see cmd/testdata/*_input.yaml for examples, and cmd/testdata/*_output.yaml for results after using kueueleuleu).

Using the library

First, run go get github.com/norbjd/kueueleuleu@main to download the dependency.

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/norbjd/kueueleuleu"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

func main() {
    // original pod: containers will run at the same time
    twoStepsPod := corev1.Pod{
        ObjectMeta: metav1.ObjectMeta{
            Name: "two-steps-pod",
        },
        Spec: corev1.PodSpec{
            Containers: []corev1.Container{
                {
                    Name:    "step1",
                    Image:   "alpine",
                    Command: []string{"sh", "-c"},
                    Args:    []string{`echo "start step1" && sleep 5 && echo "end step1"`},
                },
                {
                    Name:    "step2",
                    Image:   "alpine",
                    Command: []string{"sh", "-c"},
                    Args:    []string{`echo "start step2" && sleep 2 && echo "end step2"`},
                },
            },
            RestartPolicy: "Never",
        },
    }

    // convert the pod to run containers sequentially
    twoSequentialStepsPod, err := kueueleuleu.ConvertPod(twoStepsPod)
    if err != nil {
        panic(err)
    }

    // create the pod
    config, _ := clientcmd.BuildConfigFromFlags("", os.Getenv("KUBECONFIG"))
    kubeClient, _ := kubernetes.NewForConfig(config)

    createdPod, err := kubeClient.CoreV1().Pods("default").Create(
        context.Background(), &twoSequentialStepsPod, metav1.CreateOptions{},
    )
    if err != nil {
        panic(err)
    }

    fmt.Printf("Created: %s\n", createdPod.Name)
}

The main advantage of kueueleuleu is that you can continue to manipulate standard kubernetes resources (like Pods) in your code. There is only a single helper if you want to know which container inside the pod is really running (as now they are running sequentially):

currentlyRunningContainerName, _ := kueueleuleu.GetRunningContainerName(*createdPod)

fmt.Printf("Running container: %s\n", currentlyRunningContainerName)

As for the CLI, the conversion also work with Jobs and CronJobs: just use kueueleuleu.ConvertJob or kueueleuleu.ConvertCronJob.

Internals

Under the hood, containers sequential orchestration is managed using Tekton entrypoint. I have just "reverse-engineered" the way Tekton generates Pods from PipelineRun. But, unlike Tekton, there is no need to install a separate controller in the cluster or using CRDs, which makes kueueleuleu lighter to use. In return, kueueleuleu cannot be used for complex workflows, and I don't consider supporting these: its only job is to run containers sequentially.

I have used Tekton entrypoint because it is already doing the job, and I didn't want to rewrite another wrapper to do more or less the same thing.

Another solution I have considered had been to convert containers to init containers (see "Use init containers" section above). But, considering init containers limitations, I thought this solution was not viable.

Limitations

Containers passed in objects (Pod, Job, CronJob) MUST have their command field set. Otherwise, kueueleuleu will return an error (container does not have a command). This is because Tekton entrypoint used under the hood requires a command. Tekton Pipelines (who also uses this entrypoint) manages to "guess" the entrypoint if a command is not provided (entrypoint hack refered here). But, I'd rather not implement this here today as it does not always work (e.g. if image is not pushed into a registry, like in kind environments, we will fail to guess the entrypoint).

Documentation

Index

• Variables
• func ConvertCronJob(cronjob batchv1.CronJob) (batchv1.CronJob, error)
• func ConvertJob(job batchv1.Job) (batchv1.Job, error)
• func ConvertPod(pod corev1.Pod) (corev1.Pod, error)
• func GetRunningContainerName(pod corev1.Pod) (string, error)
• func IsKueueleuleu(meta metav1.ObjectMeta) bool

Variables

var (
	ErrNotAKueueleuleuPod               = errors.New("not a kueueleuleu pod")
	ErrPodIsSucceeded                   = errors.New("pod is succeeded")
	ErrPodIsFailed                      = errors.New("pod is failed")
	ErrPodIsNotRunning                  = errors.New("pod is not running")
	ErrSentinelAllContainersAreFinished = errors.New("all containers are finished")
)

var ErrContainerDoesNotHaveACommand = errors.New("container does not have a command, but we expect one")

Functions

func ConvertCronJob

func ConvertCronJob(cronjob batchv1.CronJob) (batchv1.CronJob, error)

func ConvertJob

func ConvertJob(job batchv1.Job) (batchv1.Job, error)

func ConvertPod

func ConvertPod(pod corev1.Pod) (corev1.Pod, error)

func GetRunningContainerName

func GetRunningContainerName(pod corev1.Pod) (string, error)

func IsKueueleuleu

func IsKueueleuleu(meta metav1.ObjectMeta) bool