README ¶
Special Resource Operator
The Special Resource Operator (SRO) manages the deployment of software stacks for hardware accelerators on an existing (day 2) OpenShift or Kubernetes cluster. SRO can be used for a case as simple as building and loading a single kernel module, or as complex as deploying the driver, device plugin, and monitoring stack for a hardware accelerator.
For loading kernel modules, SRO is designed around the use of "driver containers." Driver containers are increasingly being used in cloud-native environments, especially when run on pure container operating systems to deliver hardware drivers to the host.
Driver containers extend the kernel stack beyond the out-of-box software and hardware features of a specific kernel. Driver containers work on various container capable Linux distributions. With driver containers the host stays "clean" and there will not be any clash between different library versions or binaries on the host.
Installation
Note: The Special Resource operator has a dependency on the Node Feature Discovery (NFD) operator. If deploying on OpenShift from OperatorHub, NFD will be installed automatically. If deploying from the CLI, first install NFD.
From OperatorHub
The Special Resource Operator is available as a community operator on OperatorHub, and as an official Red Hat operator (tech-preview) starting in OpenShift 4.9.
From the CLI
Deploy to vanilla k8s:
$ git clone https://github.com/openshift-psap/special-resource-operator
$ cd special-resource-operator
$ make deploy TAG=master
Deploy to OCP:
$ git clone https://github.com/openshift-psap/special-resource-operator
$ cd special-resource-operator
$ make deploy TAG=master
To build and deploy using a custom operator image:
$ make local-image-build
$ make local-image-push
$ make deploy
Note: The image TAG will default to the name of the current git branch, but can be overriden by setting the TAG variable. See the Makefile
for more details.
To deploy the simple-kmod example special resource on OpenShift 4.x:
$ oc apply -f charts/example/simple-kmod-0.0.1/simple-kmod.yaml
Running locally
NOTE: SRO depends on objects that are created using other deployment methods. It is recommended to first deploy SRO, scale it down and then run locally.
Special Resource Operator can be run locally against a cluster using following command:
$ make deploy
$ kubectl scale -n special-resource-operator deployment/special-resource-controller-manager --replicas=0
$ make manager helm-plugins
$ HELM_PLUGINS=$PWD/helm-plugins KUBECONFIG=$HOME/.kube/config OPERATOR_NAMESPACE=special-resource-operator ./manager
SRO manages a subdirectory inside Go's os.UserCacheDir
for the Helm cache.
Creating a special resource recipe
For most updated examples and use cases please see docs/lab/README.md.
See docs/recipes.md for instructions on how to create a recipe for SRO to manage.
See charts/example
for some examples. In particular:
- The simple-kmod example shows how to build and deploy two simple kernel modules in a driver container on OpenShift.
- The centos-simple-kmod example uses the same kernel module as simple-kmod, but is written for running on a vanilla kubernetes cluster with CentOS worker nodes.
Node Feature Discovery dependency
There is a general problem when trying to configure a cluster with a special resource. One does not know which nodes have a special resource and which do not. To address this, SRO relies on the NFD operator. NFD will label the host with node specific attributes, like PCI cards, kernel or OS version and more. The .yaml template files in a special resource recipe can use these NFD labels in their nodeSelector fields to ensure that the software stack is run only on the nodes with the hardware feature. See upstream NFD for more info.
Documentation ¶
There is no documentation for this package.
Directories ¶
Path | Synopsis |
---|---|
api
|
|
v1beta1
Package v1beta1 contains API Schema definitions for the sro v1beta1 API group +kubebuilder:object:generate=true +groupName=sro.openshift.io
|
Package v1beta1 contains API Schema definitions for the sro v1beta1 API group +kubebuilder:object:generate=true +groupName=sro.openshift.io |
cmd
|
|
internal
|
|
pkg
|
|
assets
Package assets is a generated GoMock package.
|
Package assets is a generated GoMock package. |
clients
Package clients is a generated GoMock package.
|
Package clients is a generated GoMock package. |
cluster
Package cluster is a generated GoMock package.
|
Package cluster is a generated GoMock package. |
kernel
Package kernel is a generated GoMock package.
|
Package kernel is a generated GoMock package. |
lifecycle
Package lifecycle is a generated GoMock package.
|
Package lifecycle is a generated GoMock package. |
metrics
Package metrics is a generated GoMock package.
|
Package metrics is a generated GoMock package. |
poll
Package poll is a generated GoMock package.
|
Package poll is a generated GoMock package. |
proxy
Package proxy is a generated GoMock package.
|
Package proxy is a generated GoMock package. |
registry
Package registry is a generated GoMock package.
|
Package registry is a generated GoMock package. |
resource
Package resource is a generated GoMock package.
|
Package resource is a generated GoMock package. |
runtime
Package runtime is a generated GoMock package.
|
Package runtime is a generated GoMock package. |
storage
Package storage is a generated GoMock package.
|
Package storage is a generated GoMock package. |
upgrade
Package upgrade is a generated GoMock package.
|
Package upgrade is a generated GoMock package. |
test
|
|