lumigo-kubernetes-operator

README

Lumigo Kubernetes Operator

The Kubernetes operator of Lumigo provides a one-click solution to monitoring Kubernetes clusters with Lumigo.

Setup

Installation

Install the Lumigo Kubernetes operator in your Kubernets cluster with helm:

helm repo add lumigo https://lumigo-io.github.io/lumigo-kubernetes-operator
helm install lumigo lumigo/lumigo-operator --namespace lumigo-system --create-namespace

You can customize the namespace name to use something other than lumigo-system, but this will make the rest of the instructions subtly wrong :-)

You can verify that the Lumigo Kubernetes operator is up and running with:

$ kubectl get pods -n lumigo-system
NAME                                                         READY   STATUS    RESTARTS   AGE
lumigo-lumigo-operator-controller-manager-7fc8f67bcc-ffh5k   2/2     Running   0          56s

Note: While installing the Lumigo Kubernetes operator via kustomize is generally expected to work (except the uninstallation of instrumentation on removal), it is not actually supported[^1].

Upgrading

To upgrade to a newer version of the Lumigo Kubernetes operator, run:

helm repo update
helm upgrade lumigo lumigo/lumigo-operator --namespace lumigo-system

Enabling automatic tracing

Supported resource types

The Lumigo Kubernetes operator automatically adds distributed tracing to pods created via:

• Deployments (apps/v1.Deployment)
• Daemonsets (apps/v1.DaemonSet)
• ReplicaSets (apps/v1.ReplicaSet)
• StatefulSets (apps/v1.StatefulSet)
• CronJobs (batch/v1.CronJob)
• Jobs (batch/v1.Job)

The distributed tracing is provided by the Lumigo OpenTelemetry distribution for Python and Lumigo OpenTelemetry distribution for JS.

The Lumigo Kubernetes operator will automatically trace all Python and Node.js processes found in the containers of pods created in the namespaces that Lumigo traces. To activate automatic tracing for resources in a namespace, create in that namespace a Kubernetes secret containing your Lumigo token, and reference it from a Lumigo (operator.lumigo.io/v1alpha1.Lumigo) custom resource:

apiVersion: v1
kind: Secret
metadata:
  name: lumigo-credentials
stringData:
  # Kubectl won't allow you to deploy this dangling anchor.
  # Get the actual value from Lumigo following this documentation: https://docs.lumigo.io/docs/lumigo-tokens
  token: *lumigo-token # Example: t_123456789012345678901
---
apiVersion: operator.lumigo.io/v1alpha1
kind: Lumigo
metadata:
  labels:
    app.kubernetes.io/name: lumigo
    app.kubernetes.io/instance: lumigo
    app.kubernetes.io/part-of: lumigo-operator
  name: lumigo
spec:
  lumigoToken:
    secretRef:
      name: lumigo-credentials # This must match the name of the secret; the secret must be in the same namespace as this Lumigo custom resource
      key: token # This must match the key in the Kubernetes secret

Each Lumigo resource keeps in its state a list of resources it currently instruments:

$ kubectl describe lumigo -n my-namespace
Name:         lumigo
Namespace:    my-namespace
API Version:  operator.lumigo.io/v1alpha1
Kind:         Lumigo
Metadata:
  ... # Data removed for readability
Spec:
  ... # Data removed for readability
Status:
  Conditions:
  ... # Data removed for readability
  Instrumented Resources:
    API Version:       apps/v1
    Kind:              StatefulSet
    Name:              my-statefulset
    Namespace:         my-namespace
    Resource Version:  320123
    UID:               93d6d809-ac2a-43a9-bc07-f0d4e314efcc

Opting out for specific resources

To prevent the Lumigo Kubernetes operator from injecting tracing to pods managed by some resource in a namespace that contains a Lumigo resource, add the lumigo.auto-trace label set to false:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello-node
    lumigo.auto-trace: "false"  # <-- No injection will take place
  name: hello-node
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: hello-node
  template:
    metadata:
      labels:
        app: hello-node
    spec:
      containers:
      - command:
        - /agnhost
        - netexec
        - --http-port=8080
        image: registry.k8s.io/e2e-test-images/agnhost:2.39
        name: agnhost

In the logs of the Lumigo Kubernetes operator, you will see a message like the following:

1.67534267851615e+09    DEBUG   controller-runtime.webhook.webhooks   wrote response   {"webhook": "/v1alpha1/inject", "code": 200, "reason": "the resource has the 'lumigo.auto-trace' label set to 'false'; resource will not be mutated", "UID": "6d341941-c47b-4245-8814-1913cee6719f", "allowed": true}

Settings

Inject existing resources

By default, when detecting a new Lumigo resource in a namespace, the Lumigo controller will instrument existing resources of the supported types. The injection will cause new pods to be created for daemonsets, deployments, replicasets, statefulsets and jobs; cronjobs will spawn injected pods at the next iteration. To turn off the automatic injection of existing resources, create the Lumigo resource as follows

apiVersion: operator.lumigo.io/v1alpha1
kind: Lumigo
metadata:
  labels:
    app.kubernetes.io/name: lumigo
    app.kubernetes.io/instance: lumigo
    app.kubernetes.io/part-of: lumigo-operator
  name: lumigo
spec:
  lumigoToken: ...
  tracing:
    injection:
      injectLumigoIntoExistingResourcesOnCreation: false # Default: true

Remove injection from existing resources

By default, when detecting the deletion of the Lumigo resource in a namespace, the Lumigo controller will remove instrumentation from existing resources of the supported types. The injection will cause new pods to be created for daemonsets, deployments, replicasets, statefulsets and jobs; cronjobs will spawn non-injected pods at the next iteration. To turn off the automatic removal of injection from existing resources, create the Lumigo resource as follows

apiVersion: operator.lumigo.io/v1alpha1
kind: Lumigo
metadata:
  labels:
    app.kubernetes.io/name: lumigo
    app.kubernetes.io/instance: lumigo
    app.kubernetes.io/part-of: lumigo-operator
  name: lumigo
spec:
  lumigoToken: ...
  tracing:
    injection:
      removeLumigoFromResourcesOnDeletion: false # Default: true

Note: The removal of injection from existing resources does not occur on uninstallation of the Lumigo Kubernetes operator, as the role-based access control is has likely already been deleted.

Collection of Kubernetes objects

The Lumigo Kubernetes operator will automatically collect Kubernetes object versions in the namespaces with a Lumigo resource in active state, and send them to Lumigo for issue detection (e.g., when you pods crash). The collected object types are: corev1.Events, corev1.Pods, appsv1.Deployments, apps/v1.DaemonSet, apps/v1.ReplicaSet, apps/v1.StatefulSet, batch/v1.CronJob, and batch/v1.Job. Besides events, the object versions, e.g., pods, replicasets and deployments, are needed to be able to correlate events across the owner-reference chain, e.g., the pod belongs to that replicaset, which belongs to that deployment.

To disable the automated collection of Kubernetes events and object versions, you can configure your Lumigo resources as follows:

apiVersion: operator.lumigo.io/v1alpha1
kind: Lumigo
metadata:
  labels:
    app.kubernetes.io/name: lumigo
    app.kubernetes.io/instance: lumigo
    app.kubernetes.io/part-of: lumigo-operator
  name: lumigo
spec:
  lumigoToken: ...
  infrastructure:
    kubeEvents:
      enabled: false # Default: true

When a Lumigo resource is deleted from a namespace, the collection of Kubernetes events and object versions is automatically halted.

Uninstall

The removal of the Lumigo Kubernetes operator is performed by:

helm delete lumigo --namespace lumigo-system

In namespaces with the Lumigo resource having spec.tracing.injection.enabled and spec.tracing.injection.removeLumigoFromResourcesOnDeletion both set to true, supported resources that have been injected by the Lumigo Kubernetes operator will be updated to remove the injection, with the following caveat:

Note: The removal of injection from existing resources does not apply to batchv1.Job resources, as their corev1.PodSpec is immutable after the batchv1.Job resource has been created.

TLS certificates

The Lumigo Kubernetes operator injector webhook uses a self-signed certificate that is automatically generate during the installation of the Helm chart. The generated certificate has a 365 days expiration, and a new certificate will be generated every time you upgrade Lumigo Kubernetes operator's helm chart.

Events

The Lumigo Kubernetes operator will add events to the resources it instruments with the following reasons and in the following cases:

Reason	Created on resource types	Under which conditions
LumigoAddedInstrumentation	apps/v1.Deployment, apps/v1.DaemonSet, apps/v1.ReplicaSet, apps/v1.StatefulSet, batch/v1.CronJob	If a Lumigo resources exists in the namespace, and the resource is instrumented with Lumigo as a result
LumigoCannotAddInstrumentation	apps/v1.Deployment, apps/v1.DaemonSet, apps/v1.ReplicaSet, apps/v1.StatefulSet, batch/v1.CronJob	If a Lumigo resources exists in the namespace, and the resource should be instrumented by Lumigo as a result, but an error occurs
LumigoUpdatedInstrumentation	apps/v1.Deployment, apps/v1.DaemonSet, apps/v1.ReplicaSet, apps/v1.StatefulSet, batch/v1.CronJob	If a Lumigo resources exists in the namespace, and the resource has the Lumigo instrumented updated as a result
LumigoCannotUpdateInstrumentation	apps/v1.Deployment, apps/v1.DaemonSet, apps/v1.ReplicaSet, apps/v1.StatefulSet, batch/v1.CronJob	If a Lumigo resources exists in the namespace, and the resource should have the Lumigo instrumented updated as a result, but an error occurs
LumigoRemovedInstrumentation	apps/v1.Deployment, apps/v1.DaemonSet, apps/v1.ReplicaSet, apps/v1.StatefulSet, batch/v1.CronJob	If a Lumigo resources is deleted from the namespace, and the resource has the Lumigo instrumented removed as a result
LumigoCannotRemoveInstrumentation	apps/v1.Deployment, apps/v1.DaemonSet, apps/v1.ReplicaSet, apps/v1.StatefulSet, batch/v1.CronJob	If a Lumigo resources is deleted from the namespace, and the resource should have the Lumigo instrumented removed as a result, but an error occurs

[^1]: The user experience of having to install Cert Manager is unnecessarily complex, and Kustomize layers, while they may be fine for one's own applications, are simply unsound for a batteries-included, rapidly-evolving product like the Lumigo Kubernetes operator. Specifically, please expect your Kustomize layers to stop working with any release of the Lumigo Kubernetes operator.