pvclabel

README

PVC Label Sync Controller

The PVC Label Sync Controller is responsible for syncing labels set on Kubernetes PVCs to the StorageOS volume objects.

Labels are initially set at creation time by a custom CSI Provisioner, that adds PVC labels to the CSI CreateVolume parameters. These are added on top of any default parameters set in the StorageClass parameters.

This controller ensures that any PVC label changes are applied.

Some StorageOS functionality, such as setting the number of desired replicas, is done by setting or changing the storageos.com/replicas=N label on a Kubernetes PVC (where N is from 0-6). This controller ensures that the behaviour is applied to the StorageOS volume after it has been created.

Other labels, such as storageos.com/nocache and storageos.com/nocompress can only be set when the volume is created, so the PVC Label Sync Controller ignores them.

See StorageOS Feature Labels for more information.

StorageClass Defaults

Cluster administrators may set defaults for volumes by setting feature labels as parameters in the StorageClass. The PVC Label Sync Controller will load the StorageClass parameters prior to applying any label changes to ensure that they are taken into account and not removed.

The controller needs to ensure that the defaults set in the StorageClass have not changed since the volume was provisioned. Otherwise a change to a feature label in the StorageClass would get applied to all volumes it provisioned, which may not be the expected behaviour.

Since StorageClasses are immutable, changing a parameter requires deleting and recreating the StorageClass. To detect this, when the PVC is created, the UID of the StorageClass is set in the storageos.com/storageclass annotation on the PVC by the PVC StorageClass Annotation Mutator. The PVC Label Sync Controller verifies that the current StorageClass UID matches. If not, labels are not synchronised for the PVC.

To re-enable PVC label sync when there is a StorageClass UID mismatch, manually confirm that any StorageClass parameter changes are intended to be applied, then remove the PVC StorageClass annotation.

If the PVC does not have the storageos.com/storageclass annotation and was provisioned by StorageOS, the PVC Label Sync Controller will add it, using the UID of the current StorageClass matching the name. This allows PVCs created prior to v2.4.0 (when the PVC StorageClass Annotation Mutator was added), to participate in PVC Label Sync.

It is possible that pre-v2.4.0 PVCs were created with a different StorageClass than the current, and that the parameters from the new StorageClass will be applied when a label sync for the PVC is triggered.

Trigger

The controller reconcile will trigger on any Kubernetes PVC label update event where the PVC has the StorageOS CSI driver listed in the storage provisioner annotation. Specifically, PVCs must have the annotation:

volume.beta.kubernetes.io/storage-provisioner: csi.storageos.com

The annotation is added by Kubernetes when the PVC is evaluated to determine the provisioner to use. This is determined by the PVC's StorageClassName parameter, or if not set, the default StorageClass. Once set it is not changed or removed.

Reconcile

When the labels on a Kubernetes PVC with the StorageOS provisioner annotation is updated, a request is made to the StorageOS API to re-apply the labels to the corresponding StorageOS volume.

Labels prefixed with storageos.com/ have special meaning, and will likely be applied with a discrete call to the StorageOS API. This ensures that the behaviour can be applied in a strongly-consistent manner or return an error.

Remaining labels without the storageos.com/ prefix will be applied as a single API call. They have no internal meaning to StorageOS but they can be used to influence placement decisions.

If a PVC label sync fails, it will be requeued and retried after a backoff period. It is possible that the application of only a partial set of labels will succeed. If StorageOS can't apply a certain behaviour change (for example, if the change would result in a volume going offline), then only that behaviour change would fail and the remaining changes would be attempted. If any change fails, the whole set of labels will be retried until they all succeed.

Resync

In case a PVC label update event was missed during a restart or outage, a resync runs periodically. It re-applies the set of Kubernetes PVC labels to StorageOS volumes.

PVC label resync is run every hour by default (configurable via the -pvc-label-resync-interval flag). It can be disabled by setting -pvc-label-resync-interval to 0s.

Resync is run on startup after a delay defined by the -pvc-label-resync-delay flag.

Disabling

The PVC Label Sync Controller can be disabled by setting the -enable-pvc-label-sync=false flag.

Documentation

Index

• type Controller
  • func NewController(k8s client.Client, api VolumeLabeller, scheme *runtime.Scheme, log logr.Logger) (*Controller, error)
  • func (c Controller) Delete(ctx context.Context, obj client.Object) error
  • func (c Controller) Diff(ctx context.Context, objs []client.Object) ([]client.Object, error)
  • func (c Controller) Ensure(ctx context.Context, obj client.Object) error
• type Predicate
  • func (p Predicate) Update(e event.UpdateEvent) bool
• type Reconciler
  • func NewReconciler(api VolumeLabeller, k8s client.Client, resyncDelay time.Duration, ...) *Reconciler
  • func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, workers int) error
• type VolumeLabeller

Types

type Controller

type Controller struct {
	client.Client
	// contains filtered or unexported fields
}

Controller implements the Sync contoller interface, applying PVC labels to StorageOS volumes.

func NewController

func NewController(k8s client.Client, api VolumeLabeller, scheme *runtime.Scheme, log logr.Logger) (*Controller, error)

NewController returns a Controller that implements PVC label sync in StorageOS.

func (Controller) Delete

func (c Controller) Delete(ctx context.Context, obj client.Object) error

Delete is a no-op. Volume removal is handled via CSI.

func (Controller) Diff

func (c Controller) Diff(ctx context.Context, objs []client.Object) ([]client.Object, error)

Diff takes a list of Kubernets PVC objects and returns them if they exist as volumes within StorageOS but the labels are different.

func (Controller) Ensure

func (c Controller) Ensure(ctx context.Context, obj client.Object) error

Ensure applies labels set on the k8s PVC to the StorageOS volume.

StorageOS reserved labels are validated and applied first, then the remaining unreserved labels are applied.

Any errors will result in a requeue, with standard back-off retries.

There is no label sync from StorageOS to Kubernetes. This is intentional to ensure a simple flow of desired state set by users in Kubernetes to actual state set on the StorageOS volume.

type Predicate

type Predicate struct {
	predicate.IgnoreFuncs
	// contains filtered or unexported fields
}

Predicate filters events before enqueuing the keys. Ignore all but Update events, and then filter out events from non-StorageOS PVCs. Trigger a resync when labels have changed.

We don't need to react to PVC create events as PVC labels will be set in the CSI create volume request as params. This is a customization made to the CSI Provisioner.

func (Predicate) Update

func (p Predicate) Update(e event.UpdateEvent) bool

Update determines whether an object update should trigger a reconcile.

type Reconciler

type Reconciler struct {
	client.Client

	msyncv1.Reconciler
	// contains filtered or unexported fields
}

Reconciler reconciles a PVC by applying labels from the Kubernetes PVC to the StorageOS volume object.

func NewReconciler

func NewReconciler(api VolumeLabeller, k8s client.Client, resyncDelay time.Duration, resyncInterval time.Duration) *Reconciler

NewReconciler returns a new PVC label reconciler.

The resyncInterval determines how often the periodic resync operation should be run.

func (*Reconciler) SetupWithManager

func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, workers int) error

SetupWithManager registers the controller with the controller manager.

type VolumeLabeller

type VolumeLabeller interface {
	EnsureVolumeLabels(ctx context.Context, key client.ObjectKey, labels map[string]string) error
	VolumeObjects(ctx context.Context) (map[client.ObjectKey]storageos.Object, error)
}

VolumeLabeller provides access to update volume labels.