velero-plugin-for-csi

README

Velero CSI plugins

This repository contains Velero plugins for snapshotting CSI backed PVCs using the CSI beta snapshot APIs.

CSI plugin reaches GA for AWS and Azure environments as of the Velero 1.9 release. It should also work for those environments in which CSI plugin provides durable snapshot (please check WARNING section for more details), but Velero team doesn't have enough resources to verify, so please do enough tests before using CSI plugin in some environments. For those environments that don't have durable snapshot capability, CSI plugin is still in beta state. After Velero's data mover function is ready (hopefully in Velero v1.10), CSI plugin will GA for those environments too.

For a list of prerequisites and installation instructions, please refer to our documentation here.

WARNING

CSI Snapshots are a standard Kubernetes mechanism for taking snapshots. The actual implementation of snapshots varies by storage vendor. For disaster recovery, snapshots must be stored in a durable store, such as an S3 bucket, tape library, etc. and not just on the primary storage. If the snapshot is only stored on the primary storage and the storage is corrupted or destroyed the backup will be lost.

CSI snapshots on AWS EBS, Azure managed disks and Google Cloud Persistent Disk are durable and can be safely used for backup.

For all other storage systems, please check with your storage vendor. If your storage vendor doesn't support durable snapshot storage you may want to consider Velero's Restic Integration

Verified environment

Velero CSI plugin v0.4 is tested with Velero v1.10 on the following environments:

• EKS: ebs.csi.aws.com driver is tested with helm chart version helm-chart-aws-ebs-csi-driver-2.6.8 and external-snapshotter version is v5.0.1. v6.0.1 external-snapshotter is also tested with EBS CSI driver helm chart version helm-chart-aws-ebs-csi-driver-2.6.10.
• AKS: disk.csi.azure.com driver is tested. CSI related controllers are installed in control plane, so cannot find the exactly version. Tested AKS versions are 1.23.3 and v1.22.6.
• GKE: pd.csi.storage.gke.io driver is tested. CSI related controllers are installed in control plane, so cannot find the exactly version. Tested GKE versions are 1.23.8-gke.1900 and 1.25.1-gke.500.

Compatibility

Below is a listing of plugin versions and respective Velero versions that are compatible.

Plugin Version	Velero Version
v0.6.x	v1.12.x
v0.5.x	v1.11.x
v0.4.x	v1.10.x
v0.3.0	v1.9.x
v0.2.0	v1.7.x, v1.8.x

Choosing VolumeSnapshotClass For snapshotting (>=0.6.0)

Default Behavior

You can simply create a VolumeSnapshotClass for a particular driver and put a label on it to indicate that it is the default VolumeSnapshotClass for that driver. For example, if you want to create a VolumeSnapshotClass for the CSI driver disk.csi.cloud.com for taking snapshots of disks created with disk.csi.cloud.com based storage classes, you can create a VolumeSnapshotClass like this:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: test-snapclass
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: disk.csi.cloud.com

Note: For each driver type, there should only be 1 VolumeSnapshotClass with the label velero.io/csi-volumesnapshot-class: "true".

Choose VolumeSnapshotClass for a particular Backup Or Schedule

If you want to use a particular VolumeSnapshotClass for a particular backup or schedule, you can add a annotation to the backup or schedule to indicate which VolumeSnapshotClass to use. For example, if you want to use the VolumeSnapshotClass test-snapclass for a particular backup for snapshotting PVCs of disk.csi.cloud.com, you can create a backup like this:

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: test-backup
  annotations:
    velero.io/csi-volumesnapshot-class/disk.csi.cloud.com: "test-snapclass"
spec:
    includedNamespaces:
    - default

Note: Please ensure all your annotations are in lowercase. And follow the following format: velero.io/csi-volumesnapshot-class/<driver name> = <VolumeSnapshotClass Name>

Choosing VolumeSnapshotClass for a particular PVC

If you want to use a particular VolumeSnapshotClass for a particular PVC, you can add a annotation to the PVC to indicate which VolumeSnapshotClass to use. This overrides any annotation added to backup or schedule. For example, if you want to use the VolumeSnapshotClass test-snapclass for a particular PVC, you can create a PVC like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
  annotations:
    velero.io/csi-volumesnapshot-class: "test-snapclass"
spec:
    accessModes:
    - ReadWriteOnce
    resources:
        requests:
        storage: 1Gi
    storageClassName: disk.csi.cloud.com

Note: Please ensure all your annotations are in lowercase. And follow the following format: velero.io/csi-volumesnapshot-class = <VolumeSnapshotClass Name>

Filing issues

If you would like to file a GitHub issue for the plugin, please open the issue on the core Velero repo

Kinds of Plugins Included

PVCBackupItemAction

A plugin of type BackupItemAction that backs up PersistentVolumeClaims which are backed by CSI volumes.

This plugin will create a CSI VolumeSnapshot which in turn triggers the CSI driver to perform the snapshot operation on the volume.

VolumeSnapshotBackupItemAction

A plugin of type BackupItemAction that backs up volumesnapshots.snapshot.storage.k8s.io.

When invoked, this plugin will capture information about the underlying volumesnapshotcontent.snapshot.storage.k8s.io in the annotations of the volumesnapshots being backed up. This plugin will also return the underlying volumesnapshotcontent.snapshot.storage.k8s.io and the associated snapshot.storage.k8s.io.volumesnapshotclasses as additional resources to be backed up.

VolumeSnapshotContentBackupItemAction

A plugin of type BackupItemAction that backs up volumesnapshotcontent.snapshot.storage.k8s.io.

This plugin will look for snapshot delete operation secrets from the annotations on the VolumeSnapshotContent object being backed up.

VolumeSnapshotClassBackupItemAction

A plugin of type BackupItemAction that backs up snapshot.storage.k8s.io.volumesnapshotclasses.

This plugin will look for snapshot list operation secret from the annotations on the VolumeSnapshotClass object being backed up.

PVCRestoreItemAction

A plugin of type RestoreItemAction that restores PersistentVolumeClaims which were backed up by PVCBackupItemAction.

This plugin will modify the spec of the PersistentVolumeClaim being restored to use the VolumeSnapshot, created during backup, as the data source ensuring that the newly provisioned volume, to satisfy this claim, may be pre-populated using the VolumeSnapshot.

VolumeSnapshotRestoreItemAction

A plugin of type RestoreItemAction that restores volumesnapshots.snapshot.storage.k8s.io.

This plugin will use the annotations, added during backup, to create a volumesnapshotcontent.snapshot.storage.k8s.io and statically bind it to the VolumeSnapshot object being restored. The plugin will also set the necessary annotations if the original VolumeSnapshotContent had snapshot deletion secrets associated with it.

VolumeSnapshotClassRestoreItemAction

A plugin of type RestoreItemAction that restores snapshot.storage.k8s.io.volumesnapshotclasses.

This plugin will use the annotations on the object being restored to return, as additional items, any snapshot lister secret that is associated with the VolumeSnapshotClass.

Building the plugins

Official images of the plugin are available on Velero DockerHub.

For development and testing, the plugin images may be built by running the below command

$ IMAGE=<YOUR_REGISTRY>/velero-plugin-for-csi:<YOUR_TAG> make container

Known shortcomings

We are tracking known limitations with the plugins here