nodedelete

README

Node Delete Controller

The Node Delete Controller is responsible for removing nodes from the StorageOS cluster when the node has been removed from Kubernetes.

The intention is to allow StorageOS to adapt to dynamic environments where nodes are added and removed regularly, such as when spot instances are used.

Removing nodes when they are no longer required reduces load as StorageOS will no longer check for the node recovering.

Trigger

The controller reconcile will trigger on any Kubernetes Node delete event where the node has the StorageOS CSI driver annotation.

The annotation is added by the CSI node driver registrar when StorageOS starts on the node. Once added, it is not removed.

Reconcile

When a Kubernetes node with the StorageOS CSI driver annotation is deleted, a request is made to the StorageOS API to remove the node. The StorageOS API will only allow the delete to succeed if:

• The node has been marked offline (typically discovered 20-30 seconds after it goes off the network).
• The node lock has expired in etcd. Locks expire after 30 seconds.
• The node no longer holds any master deployments. Normally the master deployment would fail over to another node after it was marked offline, but in the case where no replicas have been configured, blocking the delete allows the data to be recovered. To proceed with the node removal, delete the volume first.

If the delete request failed, it will be requeued and retried after a backoff period.

Typically there will be multiple "node still in use" log entries before the node container is shutdown and StorageOS detects that it is offline. There may also be one or more "node lock has not yet expired" messages before the node is finally removed.

If the node was not found because it has already been deleted, the delete request will be considered successful.

Garbage Collection

In case a node delete event was missed during a restart or outage, a garbage collection runs periodically. It compares the list of nodes known to StorageOS, and removes any that are no longer known to Kubernetes.

Garbage collection is run every hour by default (configurable via the -node-delete-gc-interval flag). It can be disabled by setting -node-delete-gc-interval to 0s.

Garbage collection is run on startup after a delay defined by the -node-delete-gc-delay flag.

Documentation

Index

• type Controller
  • func NewController(api NodeDeleter, log logr.Logger) (*Controller, error)
  • func (c Controller) Delete(ctx context.Context, obj client.Object) error
  • func (c Controller) Ensure(ctx context.Context, obj client.Object) error
  • func (c Controller) List(ctx context.Context) ([]types.NamespacedName, error)
• type NodeDeleter
• type Predicate
  • func (p Predicate) Delete(e event.DeleteEvent) bool
• type Reconciler
  • func NewReconciler(api NodeDeleter, k8s client.Client, gcDelay time.Duration, ...) *Reconciler
  • func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, workers int) error

Types

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

Controller implements the SyncReconciler contoller interface, deleting nodes in StorageOS when they have been detected as deleted in Kubernetes.

func NewController

func NewController(api NodeDeleter, log logr.Logger) (*Controller, error)

NewController returns a Controller that implements node garbage collection in StorageOS.

func (Controller) Delete

func (c Controller) Delete(ctx context.Context, obj client.Object) error

Delete receives a k8s object that's been deleted and calls the StorageOS api to remove it from management.

func (Controller) Ensure

func (c Controller) Ensure(ctx context.Context, obj client.Object) error

Ensure is a no-op. We only care about deletes.

func (Controller) List

func (c Controller) List(ctx context.Context) ([]types.NamespacedName, error)

List returns a list of nodes known to StorageOS, as NamespacedNames. This is used for garbage collection and can be expensive. The garbage collector is run in a separate goroutine periodically, not affecting the main reconciliation control-loop.

type NodeDeleter

type NodeDeleter interface {
	DeleteNode(ctx context.Context, key client.ObjectKey) error
	ListNodes(ctx context.Context) ([]client.Object, error)
}

NodeDeleter provides access to removing nodes from StorageOS.

type Predicate

type Predicate struct {
	predicate.IgnoreFuncs
	// contains filtered or unexported fields
}

Predicate filters events before enqueuing the keys. Ignore all but Delete events, and then filter out events from non-StorageOS nodes.

func (Predicate) Delete

func (p Predicate) Delete(e event.DeleteEvent) bool

Delete determines whether an object delete should trigger a reconcile.

type Reconciler

type Reconciler struct {
	client.Client

	objectv1.Reconciler
	// contains filtered or unexported fields
}

Reconciler reconciles a Node object by deleting the StorageOS node object when the corresponding Kubernetes node is deleted.

func NewReconciler

func NewReconciler(api NodeDeleter, k8s client.Client, gcDelay time.Duration, gcInterval time.Duration) *Reconciler

NewReconciler returns a new Node delete reconciler.

The gcInterval determines how often the periodic resync operation should be run.

func (*Reconciler) SetupWithManager

func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, workers int) error

SetupWithManager registers the controller with the controller manager.