gpu_plugin

README

Intel GPU device plugin for Kubernetes

Table of Contents

• Introduction
• Modes and Configuration Options
• Operation modes for different workload types
• Installing driver and firmware for Intel GPUs
• Pre-built Images
• Installation
  • Install with NFD
  • Install with Operator
  • Verify Plugin Registration
• Testing and Demos
• Notes
  • Running GPU plugin as non-root
  • Labels created by GPU plugin
  • SR-IOV use with the plugin
  • CDI support
  • KMD and UMD
  • Health management
  • Issues with media workloads on multi-GPU setups
    • Workaround for QSV and VA-API

Introduction

Intel GPU plugin facilitates Kubernetes workload offloading by providing access to discrete (including Intel® Data Center GPU Flex & Max Series) and integrated Intel GPU devices supported by the host kernel.

Use cases include, but are not limited to:

• Media transcode
• Media analytics
• Cloud gaming
• High performance computing
• AI training and inference

For example containers with Intel media driver (and components using that), can offload video transcoding operations, and containers with the Intel OpenCL / oneAPI Level Zero backend libraries can offload compute operations to GPU.

Intel GPU plugin may register four node resources to the Kubernetes cluster:

Resource	Description
gpu.intel.com/i915	GPU instance running legacy i915 KMD
gpu.intel.com/i915_monitoring	Monitoring resource for the legacy i915 KMD devices
gpu.intel.com/xe	GPU instance running new xe KMD
gpu.intel.com/xe_monitoring	Monitoring resource for the new xe KMD devices

While GPU plugin basic operations support nodes having both (i915 and xe) KMDs on the same node, its resource management (=GAS) does not, for that node needs to have only one of the KMDs present.

For workloads on different KMDs, see KMD and UMD.

Modes and Configuration Options

Flag	Argument	Default	Meaning
-enable-monitoring	-	disabled	Enable '*_monitoring' resource that provides access to all Intel GPU devices on the node, see use
-resource-manager	-	disabled	Enable fractional resource management, see use
-health-management	-	disabled	Enable health management by requesting data from oneAPI/Level-Zero interface. Requires GPU Level-Zero sidecar. See health management
-wsl	-	disabled	Adapt plugin to run in the WSL environment. Requires GPU Level-Zero sidecar.
-shared-dev-num	int	1	Number of containers that can share the same GPU device
-allocation-policy	string	none	3 possible values: balanced, packed, none. For shared-dev-num > 1: balanced mode spreads workloads among GPU devices, packed mode fills one GPU fully before moving to next, and none selects first available device from kubelet. Default is none. Allocation policy does not have an effect when resource manager is enabled.

The plugin also accepts a number of other arguments (common to all plugins) related to logging. Please use the -h option to see the complete list of logging related options.

Operation modes for different workload types

Intel GPU-plugin supports a few different operation modes. Depending on the workloads the cluster is running, some modes make more sense than others. Below is a table that explains the differences between the modes and suggests workload types for each mode. Mode selection applies to the whole GPU plugin deployment, so it is a cluster wide decision.

Mode	Sharing	Intended workloads	Suitable for time critical workloads
shared-dev-num == 1	No, 1 container per GPU	Workloads using all GPU capacity, e.g. AI training	Yes
shared-dev-num > 1	Yes, >1 containers per GPU	(Batch) workloads using only part of GPU resources, e.g. inference, media transcode/analytics, or CPU bound GPU workloads	No
shared-dev-num > 1 && resource-management	Depends on resource requests	Any. For requirements and usage, see fractional resource management	Yes. 1000 millicores = exclusive GPU usage. See note below.

Note: Exclusive GPU usage with >=1000 millicores requires that also all other GPU containers specify (non-zero) millicores resource usage.

Installing driver and firmware for Intel GPUs

In case your host's operating system lacks support for Intel GPUs, see this page for help: Drivers for Intel GPUs

Pre-built Images

Pre-built images of this component are available on the Docker hub. These images are automatically built and uploaded to the hub from the latest main branch of this repository.

Release tagged images of the components are also available on the Docker hub, tagged with their release version numbers in the format x.y.z, corresponding to the branches and releases in this repository.

See the development guide for details if you want to deploy a customized version of the plugin.

Installation

There are multiple ways to install Intel GPU plugin to a cluster. The most common methods are described below. For alternative methods, see advanced install page.

Note: Replace <RELEASE_VERSION> with the desired release tag or main to get devel images.

Note: Add --dry-run=client -o yaml to the kubectl commands below to visualize the yaml content being applied.

Install with NFD

Deploy GPU plugin with the help of NFD (Node Feature Discovery). It detects the presence of Intel GPUs and labels them accordingly. GPU plugin's node selector is used to deploy plugin to nodes which have such a GPU label.

# Start NFD - if your cluster doesn't have NFD installed yet
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=<RELEASE_VERSION>'

# Create NodeFeatureRules for detecting GPUs on nodes
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=<RELEASE_VERSION>'

# Create GPU plugin daemonset
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/nfd_labeled_nodes?ref=<RELEASE_VERSION>'

Install with Operator

GPU plugin can be installed with the Intel Device Plugin Operator. It allows configuring GPU plugin's parameters without kustomizing the deployment files. The general installation is described in the install documentation. For configuring the GPU Custom Resource (CR), see the configuration options and operation modes.

Install alongside with GPU Aware Scheduling

GPU plugin can be installed alongside with GPU Aware Scheduling (GAS). It allows scheduling Pods which e.g. request only partial use of a GPU. The installation is described in fractional resources page.

Verify Plugin Installation

You can verify that the plugin has been installed on the expected nodes by searching for the relevant resource allocation status on the nodes:

$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.gpu\.intel\.com/i915}{'\n'}"
master
 i915: 1

Testing and Demos

The GPU plugin functionality can be verified by deploying an OpenCL image which runs clinfo outputting the GPU capabilities (detected by driver installed to the image).

1. Make the image available to the cluster:

   Build image:

   $ make intel-opencl-icd
   

   Tag and push the intel-opencl-icd image to a repository available in the cluster. Then modify the intelgpu-job.yaml's image location accordingly:

   $ docker tag intel/intel-opencl-icd:devel <repository>/intel/intel-opencl-icd:latest
   $ docker push <repository>/intel/intel-opencl-icd:latest
   $ $EDITOR ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml
   

   If you are running the demo on a single node cluster, and do not have your own registry, you can add image to node image cache instead. For example, to import docker image to containerd cache:

   $ IMAGE_NAME=opencl-icd.tar
   $ docker save -o $IMAGE_NAME intel/intel-opencl-icd:devel
   $ ctr -n=k8s.io images import $IMAGE_NAME
   $ rm $IMAGE_NAME
   

2. Create a job:

   $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml
   job.batch/intelgpu-demo-job created
   

3. Review the job's logs:

   $ kubectl get pods | fgrep intelgpu
   # substitute the 'xxxxx' below for the pod name listed in the above
   $ kubectl logs intelgpu-demo-job-xxxxx
   <log output>
   

   If the pod did not successfully launch, possibly because it could not obtain the requested GPU resource, it will be stuck in the Pending status:

   $ kubectl get pods
   NAME                      READY   STATUS    RESTARTS   AGE
   intelgpu-demo-job-xxxxx   0/1     Pending   0          8s
   

   This can be verified by checking the Events of the pod:

   $ kubectl describe pod intelgpu-demo-job-xxxxx
   ...
   Events:
     Type     Reason            Age        From               Message
     ----     ------            ----       ----               -------
     Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 Insufficient gpu.intel.com/i915.
   

Notes

Running GPU plugin as non-root

It is possible to run the GPU device plugin using a non-root user. To do this, the nodes' DAC rules must be configured to device plugin socket creation and kubelet registration. Furthermore, the deployments securityContext must be configured with appropriate runAsUser/runAsGroup.

More info: https://kubernetes.io/blog/2021/11/09/non-root-containers-and-devices/

Labels created by GPU plugin

If installed with NFD and started with resource-management, plugin will export a set of labels for the node. For detailed info, see labeling documentation.

SR-IOV use with the plugin

GPU plugin does not setup SR-IOV. It has to be configured by the cluster admin.

GPU plugin does however support provisioning Virtual Functions (VFs) to containers for a SR-IOV enabled GPU. When the plugin detects a GPU with SR-IOV VFs configured, it will only provision the VFs and leaves the PF device on the host.

CDI support

GPU plugin supports CDI to provide device details to the container. It does not yet provide any benefits compared to the traditional Kubernetes Device Plugin API. The CDI device specs will improve in the future with features that are not possible with the Device Plugin API.

To enable CDI support, container runtime has to support it. The support varies depending on the versions:

• CRI-O supports CDI by default v1.24.0 onwards.
• Containerd supports CDI from 1.7.0 onwards. 2.0.0 release will enable it by default.
• Docker supports CDI from v25 onwards.

Kubernetes CDI support is included since 1.28 release. In 1.28 it needs to be enabled via DevicePluginCDIDevices feature gate. From 1.29 onwards the feature is enabled by default.

NOTE: To use CDI outside of Kubernetes, for example with Docker or Podman, CDI specs can be generated with the Intel CDI specs generator.

KMD and UMD

There are 3 different Kernel Mode Drivers (KMD) available: i915 upstream, i915 backport and xe:

• i915 upstream is a vanilla driver that comes from the upstream kernel and is included in the common Linux distributions, like Ubuntu.
• i915 backport is an out-of-tree driver for older enterprise / LTS kernel versions, having better support for new HW before upstream kernel does. API it provides to user-space can differ from the eventual upstream version.
• xe is a new KMD that is intended to support future GPUs. While it has experimental support for latest current GPUs (starting from Tigerlake), it will not support them officially.

For optimal performance, the KMD should be paired with the same UMD variant. When creating a workload container, depending on the target hardware, the UMD packages should be selected approriately.

KMD	UMD packages	Support notes
i915 upstream	Distro Repository	For Integrated GPUs. Newer Linux kernels will introduce support for Arc, Flex or Max series.
i915 backport	Intel Repository	Best for Arc, Flex and Max series. Untested for Integrated GPUs.
xe	Source code only	Experimental support for Arc, Flex and Max series.

NOTE: Xe UMD is in active development and should be considered as experimental.

Creating a workload that would support all the different KMDs is not currently possible. Below is a table that clarifies how each domain supports different KMDs.

Domain	i915 upstream	i915 backport	xe	Notes
Compute	Default	NEO_ENABLE_i915_PRELIM_DETECTION	NEO_ENABLE_XE_DRM_DETECTION	All three KMDs can be supported at the same time.
Media	Default	ENABLE_PRODUCTION_KMD	ENABLE_XE_KMD	Xe with upstream or backport i915, not all three.
Graphics	Default	Unknown	intel-xe-kmd	i915 and xe KMDs can be supported at the same time.

Health management

Kubernetes Device Plugin API allows passing device's healthiness to Kubelet. By default GPU plugin reports all devices to be Healthy. If health management is enabled, GPU plugin retrieves health related data from oneAPI/Level-Zero interface via GPU levelzero. Depending on the data received, GPU plugin will report device to be Unhealthy if:

1. Direct health indicators report issues: memory & pci
2. Device temperature is over the limit

Temperature limit can be provided via the command line argument, default is 100C.

Issues with media workloads on multi-GPU setups

OneVPL media API, 3D and compute APIs provide device discovery functionality for applications and work fine in multi-GPU setups. VA-API and legacy QSV (MediaSDK) media APIs do not, and do not provide (e.g. environment variable) override for their default device file.

As result, media applications using VA-API or QSV, fail to locate the correct GPU device file unless it is the first ("renderD128") one, or device file name is explicitly specified with an application option.

Kubernetes device plugins expose only requested number of device files, and their naming matches host device file names (for several reasons unrelated to media). Therefore, on multi-GPU hosts, the only GPU device file mapped to the media container can differ from "renderD128", and media applications using VA-API or QSV need to be explicitly told which one to use.

These options differ from application to application. Relevant FFmpeg options are documented here:

• VA-API: https://trac.ffmpeg.org/wiki/Hardware/VAAPI
• QSV: https://github.com/Intel-Media-SDK/MediaSDK/wiki/FFmpeg-QSV-Multi-GPU-Selection-on-Linux

Workaround for QSV and VA-API

Render device shell script locates and outputs the correct device file name. It can be added to the container and used to give device file name for the application.

Use it either from another script invoking the application, or directly from the Pod YAML command line. In latter case, it can be used either to add the device file name to the end of given command line, like this:

command: ["render-device.sh", "vainfo", "--display", "drm", "--device"]

=> /usr/bin/vainfo --display drm --device /dev/dri/renderDXXX

Or inline, like this:

command: ["/bin/sh", "-c",
          "vainfo --device $(render-device.sh 1) --display drm"
         ]

If device file name is needed for multiple commands, one can use shell variable:

command: ["/bin/sh", "-c",
          "dev=$(render-device.sh 1) && vainfo --device $dev && <more commands>"
         ]

With argument N, script outputs name of the Nth suitable GPU device file, which can be used when more than one GPU resource was requested.