README
¶
Kubernetes Dynamic Local Storage with HA
This project implements a simple storage solution for kubernetes. it is based on DRBD (distribute replication block device) open source software, which is natively data redundancy and high performance.
Prerequisite
We asumed that you have all your kubernetes nodes installed lvm-utils and at least DRBD 9.0 (userspace utils and kernel module). For more about DRBD, please refer to LINBIT.
DRBD Network RAID
Userspace Apps filesystem operations get replicated to another disk on the network node, the whole process is transparent and high performance. Apps can be migrated to that secondary node quickly and seamlessly with data already have been placed right there.
DRBD Primary DRBD Secondary
+-----------------------+ +----------------------+
| Userspace | | Userspace |
| | | |
| open/read/write/close | | |
| | | | |
+--------------|--------+ +----------------------+
| Kernel | | tcp/ip | Kernel |
| +----------->---->---->--------------+ |
| | | | | |
+--------------|--------+ +-----------|----------+
| v | | v |
| Disk: /dev/drbd0 | | Disk: /dev/drbd0 |
| | | |
+-----------------------+ +----------------------+
Kubernetes meets DRBD
Kubernetes is good at stateless Pod migration, but this is not as easy for satefull Pod without dynamic storage support. So here DRBD comes to play.
- Pod mysql:5.7 runs on k8s-node-1
k8s-node-1 k8s-node-2
+--------------------------+ +---------------------------+
| Kubelet: drbd flexvolume | | Kubelete: drbd flexvolume |
|--------------------------| |---------------------------|
| Pod: mysql:5.7 | | |
| | | | |
+--------|-----------------+ +---------------------------+
| v | tcp/ip | |
| Disk: /dev/drbd0 -->-->--|------->--->---| Disk: /dev/drbd0 |
| | | | |
+--------------------------+ | +---------------------------+
DRBD Primary | DRBD Secondary
v
| k8s-node-3
| +---------------------------+
v | Kubelete: drbd flexvolume |
| |---------------------------|
| | |
v | |
| +---------------------------+
| | |
+--->--->---| Disk: /dev/drbd0 |
| |
+---------------------------+
DRBD Secondary
- Pod mysql:5.7 got killed and migrated to k8s-node-2
k8s-node-1 k8s-node-2
+--------------------------+ +---------------------------+
| Kubelet: drbd flexvolume | | Kubelete: drbd flexvolume |
|--------------------------| |---------------------------|
| | | Pod: mysql:5.7 |
| | | | |
+--------------------------+ +--------|------------------+
| | tcp/ip | v |
| Disk: /dev/drbd0 --<--<--|------<---<--- | Disk: /dev/drbd0 |
| | | | |
+--------------------------+ | +---------------------------+
DRBD Secondary | DRBD Primary
v
| k8s-node-3
| +---------------------------+
v | Kubelete: drbd flexvolume |
| |---------------------------|
| | |
v | |
| +---------------------------+
| | |
+--->--->---| Disk: /dev/drbd0 |
| |
+---------------------------+
DRBD Secondary
Limitations
Pros
- Stateless and easy to deploy
- No extra storage cluster, storage nodes co-located with kubernetes nodes
- Data natively redundancy and high performance
- Easy to use (StorageClass, PersistentVolume, PersistentVolumeClaim)
Cons
- DRBD>=9.0 can be up to 16 nodes, but only at most one primary node simultaneously
- Pod must run as single replica instance (ReadWritOnce)
- PV is available only on resource placed nodes
- Two PVs may not be used by one Pod, for their intersection nodes could be empty
Directories