README ¶
Sample CSI Driver
The purpose of this branch is to help me understand how CSI works with Kubernetest with hands on implementation.
The driver codes is mostly taken from kubernetes-csi
.
Hostpath Plugin
Ref:
- https://github.com/kubernetes-csi/drivers/tree/master/pkg/csi-common
- https://github.com/kubernetes-csi/csi-driver-host-path/
- https://github.com/kubernetes-csi/drivers
CSI
Ref: https://github.com/container-storage-interface/spec/blob/master/lib/go/csi/csi.pb.go
- csi.NewControllerClient(/grpc.ClientConn/)
- csi.NewIdentityClient(/grpcClientConn/)
- csi.NewNodeClient(/grpc.ClientConn/)
Usage:
Build Binary
$ make hostpath
Start Driver
$ sudo ./hostpathplugin --endpoint tcp://127.0.0.1:10000 -nodeid CSINode -v=5
Test using csc
Get csc
tool from https://github.com/rexray/gocsi/tree/master/csc
$ GO111MODULE=off go get -u github.com/rexray/gocsi/csc
Get plugin info
$ csc identity plugin-info --endpoint tcp://127.0.0.1:10000
Get NodeInfo
$ csc node get-info --endpoint tcp://127.0.0.1:10000
Create Volume
$ csc controller new --endpoint tcp://127.0.0.1:10000 --cap MULTI_NODE_MULTI_WRITER,mount,xfs,uid=500 <CSIVolumeName>
<CSIVolumeID> # Example: 9c4c997a-4d4c-11ea-84cd-525400663918
Publish Volume
$ csc node publish --endpoint tcp://127.0.0.1:10000 --cap MULTI_NODE_MULTI_WRITER,mount,xfs,uid=500 --target-path /mnt/hostpath <CSIVolumeID>
<CSIVolumeID>
Unpublish Volume
$ csc node unpublish --endpoint tcp://127.0.0.1:10000 --target-path /mnt/hostpath <CSIVolumeID>
<CSIVolumeID>
Delete Volume
$ csc controller del --endpoint tcp://127.0.0.1:10000 <CSIVolumeID>
<CSIVolumeID>
List Volume
$ csc controller list-volumes --endpoint tcp://127.0.0.1:10000
Kubernetes - Sidecar Containers
Driver-registrar
- register the CSI driver with kubelet
- adds the driver custom NodeID to a lable on the Kubernetes Node API Object.
- deploy
plugin
here.
External-provisioner
- watches Kubernetes PersistentVolumeClaim objects and triggers CreateVolume/DeleteVolume agains a CSI endpoint
External-attacher
- watches Kubernetes VolumeAttachment objects and triggers ControllerPublish/Unpublish against a CSI endpoint
External-snapshotter
- watches for
VolumeSnapshotContent
create/update/delete events. It filters out these objects withDriver==<CSI driver name>
specified in the associated VolumeSnapshotClass object and then processes these events in workqueues with exponential backoff.
Procedures
Pre-requisite
The volume snapshot is an alpha feature in Kubernetes v1.12 to v1.16, you may need to enable a new alpha feature gate called VolumeSnapshotDataSource in the Kubernetes master.
--feature-gates=VolumeSnapshotDataSource=true
Ref: https://kubernetes-csi.github.io/docs/snapshot-restore-feature.html
With Kuberntest v1.17 and above, external-snapshotter 2.0 and higher, CRDs are no longer deployed by the csi-snapshotter.
Install CRD If is not pre-installed in cluster. Ref: https://github.com/kubernetes-csi/external-snapshotter.
# https://github.com/kubernetes-csi/external-snapshotter/tree/master/config/crd
kubectl create -f manifest/crd
Deploy
$ kubectl create -f manifest/deploy
Verify
- Check CSI resources are
Running
.
$ kubectl -n kube-system get pods | grep csi
csi-hostpath-attacher-0 1/1 Running 0 18s
csi-hostpath-provisioner-0 1/1 Running 0 17s
csi-hostpath-registrar-0 2/2 Running 0 16s
csi-hostpath-snapshotter-0 1/1 Running 0 14s
- Deploy sample resources.
$ kubectl create -f manifest/sample
- Check sample
Persistent Volume Claim
bonded to storage class.
$ kubectl -n kube-system get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc Bound pvc-2ba7ef84-770b-40bc-b9b5-af2e7c120b39 1Gi RWO csi-hostpath-sc 8m51s
- Check sample
Pod
running.
$ kubectl -n kube-system get pod sample
NAME READY STATUS RESTARTS AGE
sample 1/1 Running 0 65s
- Create
hello.txt
file in samplePod
.
$ kubectl -n kube-system exec -it sample -- sh -c "echo 'I am here' > /data/hello.txt"
$ kubectl -n kube-system exec -it sample -- sh -c "cat /data/hello.txt"
I am here
- Check volume attached to registrar(plugin) container.
$ POD=`kubectl -n kube-system get pods --selector app=csi-hostpath-registrar -o jsonpath='{.items[*].metadata.name}'`
$ kubectl -n kube-system exec -it ${POD} -c hostpath -- sh -c "find / -name hello.txt"
/var/lib/kubelet/pods/2d2e2a9f-7789-4f28-ae5f-414f53334751/volumes/kubernetes.io~csi/pvc-2ba7ef84-770b-40bc-b9b5-af2e7c120b39/mount/hello.txt
/csi-data-dir/61281e9c-4e11-11ea-8b84-fa9b3842591d/hello.txt
- Check with
VolumeAttachment
object is created.
$ kubectl describe volumeattachment
Name: csi-dff51584a67e081d7fb91ced0d2e485ca78c27f2307357256947e1001711e922
Namespace:
Labels: <none>
Annotations: <none>
API Version: storage.k8s.io/v1
Kind: VolumeAttachment
Metadata:
Creation Timestamp: 2020-02-13T03:42:34Z
Resource Version: 186626
Self Link: /apis/storage.k8s.io/v1/volumeattachments/csi-dff51584a67e081d7fb91ced0d2e485ca78c27f2307357256947e1001711e922
UID: 4f63316d-2234-4506-a61b-f7cbf823b8f6
Spec:
Attacher: hostpath.csi.k8s.io
Node Name: caasp-worker-c3y1-cluster-0
Source:
Persistent Volume Name: pvc-2ba7ef84-770b-40bc-b9b5-af2e7c120b39
Status:
Attached: true
Events: <none>
Snapshot
- Deploy snapshot resources.
$ kubectl create -f manifest/snapshot
volumesnapshot.snapshot.storage.k8s.io/csi-hostpath-snapshot created
- Check snapshot.
$ kubectl -n kube-system get volumesnapshot
NAME AGE
csi-hostpath-snapshot 10m
$ kubectl -n kube-system get volumesnapshotcontent
NAME AGE
snapcontent-99b5c4d9-4c9e-41b0-ad15-c8797bdc133d 11m
Restore
- Delete sample resources.
$ kubectl delete -f manifest/sample
- Restore sample resources.
$ kubectl create -f manifest/sample/storage-class.yaml
$ kubectl create -f manifest/restore/persistent-volume-claim.yaml
$ kubectl create -f manifest/sample/pod.yaml
- Check
hello.txt
file exist in the restored Pod.
$ kubectl -n kube-system exec -it sample -- sh -c "cat /data/hello.txt"
I am here
QA
Where is plugin deployed to
Plugin for this example is build to single binary and deployed in registrar
.
Unfinished Business
- Go through csi-api to understand how Kubernetes implement CSI CRD.
- Go through external-provisioner, external-snapshotter, external-attacher to understand how each sidecar works.