eks-ebs-node-operator

README

EKS ebs node operator

Operator adds custom resource limit to the AWS EKS worker nodes, calculated from node type and some AWS imposed limits. At the time of writing there is a configuration mismatch between Kubernetes EBS CSI and AWS EC2 instanced imposed limitations, which in some cases leads to pods with ebs volumes being scheduled on a node, where it is impossible to attach EBS volume anymore.

When an operator is deployed in the cluster it will automatically add extra custom resource limit, but pods needs to consume this resource via resources, like CPU or Memory. Kubernetes will automatically calculate the amount of EBS volume attachments left on a node and will not schedule pods with EBS volumes if the resource is exhausted.

Limits

The limit calculation is based on:

• https://github.com/kubernetes/kubernetes/issues/80967
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html

where:

• A1, C5, C5d, C5n, I3en, M5, M5a, M5ad, M5d, p3dn.24xlarge, R5, R5a, R5ad, R5d, T3, T3a, and z1d <= 28
• 28 - 1 (root volume) - 110/interface capacity (num of interfaces) - number of NVMe volumes

The definitions can be found in pkg/controller/node/node_controller.go

Installation

Repository contains directory deploy with 2 helm charts, which are tested with helm version: 2.15.1 and 3.2.1 respectively.

Containers configuration

To consume imposed custom resource limit eksattachments, the resource request and limit must be set for each container in the cluster which attaches EBS volume as PV. Same example can be used to configure StatefulSet, DaemonSet, Deployment or ReplicaSet, for example the pod which have one container and mounts one EBS volume will have similar to following definition:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  namespace: default
spec:
  containers:
  - image: mycompany/myimage:X.Y.Z
    name: container
    ...
    resources:
      limits:
        eks.ebsnodeoperator/attachments: "1"
      requests:
        eks.ebsnodeoperator/attachments: "1"
    volumeMounts:
    - mountPath: /mydata
      name: mydata
  volumes:
  - name: mydata
    persistentVolumeClaim:
      claimName: my-data-0
  ...

Use kubectl describe node to see that pod consumed 1 custom resource.

Credits

• Yevhen Volchenko for the idea to use custom resource limits