Based on the original work of https://github.com/hustcat/k8s-rdma-device-plugin
Introduction
k8s-widget-device-plugin
is a device plugin for Kubernetes to create mock devices for scheduling testing purposes.
Quick Start
Build
Relies on vndr (https://github.com/LK4D4/vndr), which can be installed with go get github.com/LK4D4/vndr
, and must be found in $PATH.
Run build
:
# make all
# ls bin
k8s-ghost-device-plugin
Work with Kubernetes
Start kubelet
with --feature-gates=DevicePlugins=true
.
- Create a resource config file:
devices:
- type: tpm
model: v1
connection: 10.0.10.2:10000
host: minikube
- type: tpm
model: v2
connection: 10.0.10.3:10001
host: minikube
- type: tpm
model: v2
connection: 10.0.10.4:10001
host: notminikube
- type: thing
model: v2
connection: 10.0.10.64:10001
- Run device plugin daemon process
# bin/k8s-rdma-device-plugin -resource-configfile ./widget.yml -log-level debug
DEBU[0000] Config file: /etc/kubernetes/widget.yml
DEBU[0000] Resource Name: skatelescope.org/widget
INFO[0000] Fetching devices.
DEBU[0000] Going to read config
DEBU[0000] loading config from file '/etc/kubernetes/widget.yml' => {"Devices":[{"Type":"snaffler","Model":"v1","Device":"/dev/wibble1"},{"Type":"snaffler","Model":"v1","Device":"/dev/wibble2"},{"Type":"snaffler","Model":"v2","Device":"/dev/wibble3"}]}
DEBU[0000] Widget device list: [{v1 snaffler 0 snaffler_v1_0 /dev/wibble1} {v1 snaffler 1 snaffler_v1_1 /dev/wibble2} {v2 snaffler 2 snaffler_v2_2 /dev/wibble3}]
INFO[0000] Starting FS watcher.
INFO[0000] Starting OS watcher.
DEBU[0000] other instance of GetWidgetDevices
DEBU[0000] Going to read config
DEBU[0000] loading config from file '/etc/kubernetes/widget.yml' => {"Devices":[{"Type":"snaffler","Model":"v1","Device":"/dev/wibble1"},{"Type":"snaffler","Model":"v1","Device":"/dev/wibble2"},{"Type":"snaffler","Model":"v2","Device":"/dev/wibble3"}]}
DEBU[0000] Base64 encoded Resource Name: c2thLXNkcC5vcmcvd2lkZ2V0
INFO[0000] Starting to serve on /var/lib/kubelet/device-plugins/c2thLXNkcC5vcmcvd2lkZ2V0_widget.sock
INFO[0000] Registered device plugin with Kubelet
...
or deploy it as a daemonset:
# kubectl -n kube-system apply -f ghost-device-plugin.yml
# kubectl -n kube-system get pods
ghost-device-plugin-daemonset-2wbdv 1/1 Running 0 14m
ghost-device-plugin-daemonset-7pwf7 1/1 Running 0 14m
- List out the allocatable resource, and capacity per node
kubectl get nodes -o json | jq '.items[] | .metadata.name, .status.allocatable, .status.capacity'
Which gives:
"k8s-master-0"
{
"cpu": "64",
"ephemeral-storage": "337894066846",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "131813436Ki",
"pods": "110",
"skatelescope.org/widget": "0" # not scheduled on master
}
"k8s-minion-0"
{
"cpu": "64",
"ephemeral-storage": "337894066846",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "131813436Ki",
"pods": "110",
"skatelescope.org/widget": "3"
}
...
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: ghost-container
namespace: default
spec:
selector:
matchLabels:
k8s-app: ghost-container
replicas: 1
template:
metadata:
labels:
k8s-app: ghost-container
spec:
containers:
- name: ghost-container
image: busybox:latest
resources:
limits:
skatelescope.org/widget: 1 # requesting 1 ghost device
securityContext:
capabilities:
add: ["ALL"]
command:
- env;
- /bin/sleep
- "360000"
TODO
Run multiple types of mock devices