special-resource-operator

README

Special Resource Operator

The Special Resource Operator (SRO) manages the deployment of software stacks for hardware accelerators on an existing (day 2) OpenShift or Kubernetes cluster. SRO can be used for a case as simple as building and loading a single kernel module, or as complex as deploying the driver, device plugin, and monitoring stack for a hardware accelerator.

For loading kernel modules, SRO is designed around the use of "driver containers." Driver containers are increasingly being used in cloud-native environments, especially when run on pure container operating systems to deliver hardware drivers to the host.

Driver containers extend the kernel stack beyond the out-of-box software and hardware features of a specific kernel. Driver containers work on various container capable Linux distributions. With driver containers the host stays "clean" and there will not be any clash between different library versions or binaries on the host.

Installation

Note: The Special Resource operator has a dependency on the Node Feature Discovery (NFD) operator. If deploying on OpenShift from OperatorHub, NFD will be installed automatically. If deploying directly from Github (this repo), first install NFD.

From OperatorHub

The Special Resource Operator is available as a community operator on OperatorHub, and as an official Red Hat operator (tech-preview) starting in OpenShift 4.9.

From the CLI

$ git clone https://github.com/openshift/special-resource-operator
$ cd special-resource-operator
$ TAG=master make deploy

To build and deploy using a custom operator image:

$ make local-image-build
$ make local-image-push
$ make deploy

Note: The image TAG will default to the name of the current git branch, but can be overriden by setting the TAG variable. See the Makefile for more details.

To deploy the simple-kmod example special resource on OpenShift 4.x:

$ VERSION=0.0.1 REPO=example SPECIALRESOURCE=simple-kmod make

which is equivalent to

$ oc apply -f charts/example/simple-kmod-0.0.1/simple-kmod.yaml

Running locally

NOTE: SRO depends on objects that are created using other deployment methods. It is recommended to first deploy SRO, scale it down and then run locally.

Special Resource Operator can be run locally against a cluster using following command:

$ make deploy
$ oc scale -n openshift-special-resource-operator deployment/special-resource-controller-manager --replicas=0
$ make manager helm-plugins
$ HELM_PLUGINS=$PWD/helm-plugins KUBECONFIG=$HOME/.kube/config OPERATOR_NAMESPACE=openshift-special-resource-operator ./manager

SRO manages a subdirectory inside Go's os.UserCacheDir for the Helm cache.

Creating a special resource recipe

For most updated examples and use cases please see docs/lab/README.md.

See docs/recipes.md for instructions on how to create a recipe for SRO to manage.

See charts/example for some examples. In particular:

• The simple-kmod example shows how to build and deploy two simple kernel modules in a driver container on OpenShift.
• The centos-simple-kmod example uses the same kernel module as simple-kmod, but is written for running on a vanilla kubernetes cluster with CentOS worker nodes.

Creating ConfigMap with Helm Chart

Convenience script can be used to create a ConfigMap containing Helm Chart:

# following will create "simple-kmod-0.0.1" ConfigMap in "default" namespace
$ ./scripts/make-cm-recipe ./charts/example/simple-kmod-0.0.1

# name and namespace can be specified
$ ./scripts/make-cm-recipe ./charts/example/simple-kmod-0.0.1 configmap-name namespace

To use ConfigMap in Special Resource recipe use following URI scheme: cm://namespace/configmap-name, for example:

apiVersion: sro.openshift.io/v1beta1
kind: SpecialResource
metadata:
  name: simple-kmod
spec:
  chart:
    (...)
    repository:
      name: example
      url: cm://default/simple-kmod-0.0.1
  set:
    (...)

IMPORTANT: When having more than one SpecialResource, use unique spec.chart.repository.name values.

Node Feature Discovery dependency

There is a general problem when trying to configure a cluster with a special resource. One does not know which nodes have a special resource and which do not. To address this, SRO relies on the NFD operator. NFD will label the host with node specific attributes, like PCI cards, kernel or OS version and more. The .yaml template files in a special resource recipe can use these NFD labels in their nodeSelector fields to ensure that the software stack is run only on the nodes with the hardware feature. See upstream NFD for more info.