qat_plugin

README

Intel QuickAssist Technology (QAT) device plugin for Kubernetes

Table of Contents

• Introduction
  • Modes and Configuration options
• Installation
  • Pre-built image
  • Prerequisites
  • Getting the source code:
  • Verify node kubelet config
  • Deploying as a DaemonSet
    • Build the plugin image
    • Deploy the DaemonSet
    • Verify QAT device plugin is registered on master:
  • Deploying by hand
    • Build QAT device plugin
    • Deploy QAT plugin
  • QAT device plugin Demos
    • DPDK QAT demos
      • DPDK Prerequisites
      • Build the image
      • Deploy the pod
      • Manual test run
      • Automated test run
    • OpenSSL QAT demo
• Checking for hardware

Introduction

This Intel QAT device plugin provides support for Intel QAT devices under Kubernetes. The supported devices are determined by the VF device drivers available in your Linux Kernel. See the Prerequisites section for more details.

Supported Devices include, but may not be limited to, the following:

• Intel® Xeon® with Intel® C62X Series Chipset
• Intel® Atom™ Processor C3000
• Intel® Communications Chipset 8925 to 8955 Series

The QAT device plugin provides access to QAT hardware accelerated cryptographic and compression features. Demonstrations are provided utilising DPDK and OpenSSL.

Kata Containers QAT integration is documented in the Kata Containers documentation repository.

Modes and Configuration options

The QAT plugin can take a number of command line arguments, summarised in the following table:

Flag	Argument	Meaning
-debug		enable debug output
-dpdk-driver	string	DPDK Device driver for configuring the QAT device (default: vfio-pci)
-kernel-vf-drivers	string	Comma separated VF Device Driver of the QuickAssist Devices in the system. Devices supported: DH895xCC,C62x,C3xxx and D15xx (default: dh895xccvf,c6xxvf,c3xxxvf,d15xxvf)
-max-num-devices	int	maximum number of QAT devices to be provided to the QuickAssist device plugin (default: 32)
-mode	string	plugin mode which can be either dpdk or kernel (default: dpdk)

The example DaemonSet YAML passes a number of these arguments, and takes its default values from the QAT default ConfigMap. The following table summarises the defaults:

Argument	Variable	Default setting	Explanation
-debug	$DEBUG	false	Debug is disabled by default
-dpdk-driver	$DPDK_DRIVER	vfio-pci	A more robust and secure choice than the igb_uio alternative
-kernel-vf-drivers	$KERNEL_VF_DRIVERS	dh895xccvf,c6xxvf,c3xxxvf,d15xxvf	Modify to suit your hardware setup
-max-num-devices	$MAX_NUM_DEVICES	32	Modify to suit your hardware setup if necessary

For more details on the -dpdk-driver choice, see DPDK Linux Driver Guide.

For more details on the available options to the -kernel-vf-drivers option, see the list of vf drivers available in the Linux Kernel.

If the -mode parameter is set to kerneldrv, no other parameter documented above are valid, except -debug which is global for both modes. kerneldrv mode implements resource allocation based on system configured logical instances.

Note: kerneldrv mode is excluded by default from all builds (including those hosted on the Docker hub), by default. See the Build the plugin image section for more details.

The kerneldrv mode does not guarantee full device isolation between containers and therefore it's not recommended. This mode will be deprecated and removed once libqat implements non-UIO based device access.

Installation

The below sections cover how to obtain, build and install this component.

The component can be installed either using a DaemonSet or running 'by hand' on each node.

Pre-built image

Pre-built images of this component are available on the Docker hub. These images are automatically built and uploaded to the hub from the latest master branch of this repository.

Release tagged images of the components are also available on the Docker hub, tagged with their release version numbers in the format x.y.z, corresponding to the branches and releases in this repository.

The deployment YAML files supplied with the component in this repository use the images with the devel tag by default. If you do not build your own local images, your Kubernetes cluster may pull down the devel images from the Docker hub by default.

To use the release tagged versions of the images, edit the YAML deployment files appropriately.

Prerequisites

The component has the same basic dependancies as the generic plugin framework dependencies.

You will also need appropriate hardware installed.

The QAT plugin requires Linux Kernel VF QAT drivers to be available. These drivers are available via two methods. One of them must be installed and enabled:

• Linux Kernel upstream drivers
• Intel QuickAssist Technology software for Linux

The demonstrations have their own requirements, listed in their own specific sections.

Getting the source code

$ mkdir -p $(go env GOPATH)/src/github.com/intel
$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes

Verify node kubelet config

Every node that will be running the plugin must have the kubelet device-plugins configured. For each node, check that the kubelet device plugin socket exists:

$ ls /var/lib/kubelet/device-plugins/kubelet.sock
/var/lib/kubelet/device-plugins/kubelet.sock

Deploying as a DaemonSet

To deploy the plugin as a DaemonSet, you first need to build a container image for the plugin and ensure that is visible to your nodes. If you do not build your own plugin, your cluster may pull the image from the pre-built Docker Hub images, depending on your configuration.

Build the plugin image

The following will use docker to build a local container image called intel/intel-qat-plugin with the tag devel. The image build tool can be changed from the default docker by setting the BUILDER argument to the Makefile.

$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes
$ make intel-qat-plugin
...
Successfully tagged intel/intel-qat-plugin:devel

Note: kerneldrv mode is excluded from the build by default. Add BUILDTAGS=kerneldrv to make to get kerneldrv functionality added to the build.

Deploy the DaemonSet

Deploying the plugin involves first the deployment of a ConfigMap and the DaemonSet YAML.

There is a kustomization for deploying both:

$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes
$ kubectl apply -k deployments/qat_plugin

and an alternative kustomization for deploying the plugin in the debug mode:

$ kubectl apply -k deployments/qat_plugin/overlays/debug

The third option is to deploy the yamls separately:

$ kubectl create -f deployments/qat_plugin/base/intel-qat-plugin-config.yaml
$ kubectl create -f deployments/qat_plugin/base/intel-qat-plugin.yaml

Note: It is also possible to run the QAT device plugin using a non-root user. To do this, the nodes' DAC rules must be configured to allow PCI driver unbinding/binding, device plugin socket creation and kubelet registration. Furthermore, the deployments securityContext must be configured with appropriate runAsUser/runAsGroup.

Verify QAT device plugin is registered on master:

Verification of the plugin deployment and detection of QAT hardware can be confirmed by examining the resource allocations on the nodes:

$ kubectl describe node <node name> | grep qat.intel.com/generic
 qat.intel.com/generic: 10
 qat.intel.com/generic: 10

Deploying by hand

For development purposes, it is sometimes convenient to deploy the plugin 'by hand' on a node. In this case, you do not need to build the complete container image, and can build just the plugin.

Build QAT device plugin

$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes
$ make qat_plugin

Deploy QAT plugin

Deploy the plugin on a node by running it as root. The below is just an example - modify the paramaters as necessary for your setup:

$ sudo $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes/cmd/qat_plugin/qat_plugin \
-dpdk-driver igb_uio -kernel-vf-drivers dh895xccvf -max-num-devices 10 -debug
QAT device plugin started
Discovered Devices below:
03:01.0 device: corresponding DPDK device detected is uio0
03:01.1 device: corresponding DPDK device detected is uio1
03:01.2 device: corresponding DPDK device detected is uio2
03:01.3 device: corresponding DPDK device detected is uio3
03:01.4 device: corresponding DPDK device detected is uio4
03:01.5 device: corresponding DPDK device detected is uio5
03:01.6 device: corresponding DPDK device detected is uio6
03:01.7 device: corresponding DPDK device detected is uio7
03:02.0 device: corresponding DPDK device detected is uio8
03:02.1 device: corresponding DPDK device detected is uio9
The number of devices discovered are:10
device-plugin start server at: /var/lib/kubelet/device-plugins/intelQAT.sock
device-plugin registered
ListAndWatch: Sending device response

QAT device plugin Demos

The below sections cover DPDK and OpenSSL demos, both of which utilise the QAT device plugin under Kubernetes.

DPDK QAT demos

The Data Plane Development Kit (DPDK) QAT demos use DPDK crypto-perf and compress-perf utilities to exercise DPDK QAT Poll-Mode Drivers (PMD). For more information on the tools' parameters, refer to the website links.

DPDK Prerequisites

For the DPDK QAT demos to work, the DPDK drivers must be loaded and configured. For more information, refer to: DPDK Getting Started Guide for Linux and DPDK Getting Started Guide, Linux Drivers section

Build the image

The demo uses a container image. You can either use the pre-built image from the Docker Hub, or build your own local copy.

To build the DPDK demo image:

$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes/demo
$ ./build-image.sh crypto-perf
...
Successfully tagged crypto-perf:devel

Deploy the pod

In the pod specification file, add container resource request and limit. For example, qat.intel.com/generic: <number of devices> for a container requesting QAT devices.

For a DPDK-based workload, you may need to add hugepage request and limit.

$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes
$ kubectl apply -k deployments/qat_dpdk_app/base/
$ kubectl get pods
  NAME                     READY     STATUS    RESTARTS   AGE
  qat-dpdk                 1/1       Running   0          27m
  intel-qat-plugin-5zgvb   1/1       Running   0          3h

Note: The deployment example above uses kustomize that is available in kubectl since Kubernetes v1.14 release.

Manual test run

Manually execute the dpdk-test-crypto-perf application to review the logs:

$ kubectl exec -it qat-dpdk bash

$ dpdk-test-crypto-perf -l 6-7 -w $QAT1 \
-d /usr/lib64/librte_mempool_ring.so.1.1 \
-d /usr/lib64/librte_pmd_qat.so.1.1 -- \
--ptest throughput --devtype crypto_qat \
--optype cipher-only --cipher-algo aes-cbc --cipher-op encrypt \
--cipher-key-sz 16 --total-ops 10000000 --burst-sz 32 --buffer-sz 64

Note: Adapt the .so versions to what the DPDK version in the container provides.

Automated test run

It is also possible to deploy and run crypto-perf using the following kustomize overlays:

$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes
$ kubectl apply -k deployments/qat_dpdk_app/test-crypto1
$ kubectl apply -k deployments/qat_dpdk_app/test-compress1
$ kubectl logs qat-dpdk-test-crypto-perf-tc1
$ kubectl logs qat-dpdk-test-compress-perf-tc1

Note: for test-crypto1 and test-compress1 to work, the cluster must enable Kubernetes CPU manager's static policy.

OpenSSL QAT demo

Please refer to the Kata Containers documentation for details on the OpenSSL QAT acceleration demo.

Checking for hardware

In order to utilise the QAT device plugin, QuickAssist SR-IOV virtual functions must be configured. You can verify this on your nodes by checking for the relevant PCI identifiers:

for i in 0442 0443 37c9 19e3; do lspci -d 8086:$i; done