preemptiontoleration

README

Overview

This folder holds the preemption toleration plugin implemented as discussed in Preemption Toleration.

Maturity Level

• 💡 Sample (for demonstrating and inspiring purpose)
• 👶 Alpha (used in companies for pilot projects)
• 👦 Beta (used in companies and developed actively)
• 👨 Stable (used in companies for production workloads)

Example scheduler config:

apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: false
clientConnection:
  kubeconfig: "REPLACE_ME_WITH_KUBE_CONFIG_PATH"
profiles:
- schedulerName: default-scheduler
  plugins:
    postFilter:
      enabled:
      - name: PreemptionToleration
      disabled:
      - name: DefaultPreemption

How to define PreemptionToleration policy on PriorityClass resource

Preemption toleration policy can be defined on each PriorityClass resource by annotations like below:

# PriorityClass with PreemptionToleration policy:
# Any pod P in this priority class can not be preempted (can tolerate preemption)
# - by preemptor pods with priority < 10000 
# - and if P is within 1h since being scheduled
kind: PriorityClass
metadata:
  name: toleration-policy-sample
  annotation:
    preemption-toleration.scheduling.sigs.k8s.io/minimum-preemptable-priority: "10000"
    preemption-toleration.scheduling.sigs.k8s.io/toleration-seconds: "3600"
value: 8000

Documentation

Index

• Constants
• func ExemptedFromPreemption(victimCandidate, preemptor *v1.Pod, ...) (bool, error)
• func New(rawArgs runtime.Object, fh framework.Handle) (framework.Plugin, error)
• type Policy
• type PreemptionToleration
  • func (pl *PreemptionToleration) CandidatesToVictimsMap(candidates []preemption.Candidate) map[string]*extenderv1.Victims
  • func (pl *PreemptionToleration) GetOffsetAndNumCandidates(numNodes int32) (int32, int32)
  • func (pl *PreemptionToleration) Name() string
  • func (pl *PreemptionToleration) PodEligibleToPreemptOthers(pod *v1.Pod, nominatedNodeStatus *framework.Status) (bool, string)
  • func (pl *PreemptionToleration) PostFilter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, ...) (*framework.PostFilterResult, *framework.Status)
  • func (pl *PreemptionToleration) SelectVictimsOnNode(ctx context.Context, state *framework.CycleState, preemptor *v1.Pod, ...) ([]*v1.Pod, int, *framework.Status)

Constants

const (
	AnnotationKeyPrefix                     = "preemption-toleration.scheduling.sigs.k8s.io/"
	AnnotationKeyMinimumPreemptablePriority = AnnotationKeyPrefix + "minimum-preemptable-priority"
	AnnotationKeyTolerationSeconds          = AnnotationKeyPrefix + "toleration-seconds"
)

const (
	// Name of the plugin used in the plugin registry and configurations.
	Name = "PreemptionToleration"
)

Functions

func ExemptedFromPreemption

func ExemptedFromPreemption(
	victimCandidate, preemptor *v1.Pod,
	pcLister schedulinglisters.PriorityClassLister,
	now time.Time,
) (bool, error)

ExemptedFromPreemption evaluates whether the victimCandidate pod can tolerate from preemption by the preemptor pod or not by inspecting PriorityClass of victimCandidate pod. The function is public because other plugin can evaluate preemption toleration policy This would be useful other PostFilter plugin depends on the preemption toleration feature.

func New

func New(rawArgs runtime.Object, fh framework.Handle) (framework.Plugin, error)

New initializes a new plugin and returns it.

Types

type Policy

type Policy struct {
	// MinimumPreemptablePriority specifies the minimum priority value that can preempt this priority class.
	// It defaults to the PriorityClass's priority value + 1 if not set, which means pods that have a higher priority value can preempt it.
	MinimumPreemptablePriority int32

	// TolerationSeconds specifies how long this priority class can tolerate preemption
	// by priorities lower than MinimumPreemptablePriority.
	// It defaults to zero if not set. Zero value means the pod will be preempted immediately. i.e., no toleration at all.
	// If it's set to a positive value, the duration will be honored.
	// If it's set to a negative value, the pod can be tolerated forever - i.e., pods with priority
	// lower than MinimumPreemptablePriority won't be able to preempt it.
	// This value affects scheduled pods only (no effect on nominated pods).
	TolerationSeconds int64
}

Policy holds preemption toleration policy configuration. Each property values are annotated in the target PriorityClass resource. Example:

kind: PriorityClass
  metadata:
  name: toleration-policy-sample
  annotation:
    preemption-toleration.scheduling.sigs.k8s.io/minimum-preemptable-priority: "10000"
    preemption-toleration.scheduling.sigs.k8s.io/toleration-seconds: "3600"

type PreemptionToleration

type PreemptionToleration struct {
	// contains filtered or unexported fields
}

PreemptionToleration is a PostFilter plugin implements the preemption logic.

func (*PreemptionToleration) CandidatesToVictimsMap

func (pl *PreemptionToleration) CandidatesToVictimsMap(candidates []preemption.Candidate) map[string]*extenderv1.Victims

func (*PreemptionToleration) GetOffsetAndNumCandidates

func (pl *PreemptionToleration) GetOffsetAndNumCandidates(numNodes int32) (int32, int32)

GetOffsetAndNumCandidates chooses a random offset and calculates the number of candidates that should be shortlisted for dry running preemption.

func (*PreemptionToleration) Name

func (pl *PreemptionToleration) Name() string

Name returns name of the plugin. It is used in logs, etc.

func (*PreemptionToleration) PodEligibleToPreemptOthers

func (pl *PreemptionToleration) PodEligibleToPreemptOthers(pod *v1.Pod, nominatedNodeStatus *framework.Status) (bool, string)

PodEligibleToPreemptOthers determines whether this pod should be considered for preempting other pods or not. If this pod has already preempted other pods and those are in their graceful termination period, it shouldn't be considered for preemption. We look at the node that is nominated for this pod and as long as there are terminating pods on the node, we don't consider this for preempting more pods.

func (*PreemptionToleration) PostFilter

func (pl *PreemptionToleration) PostFilter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, m framework.NodeToStatusMap) (*framework.PostFilterResult, *framework.Status)

PostFilter invoked at the postFilter extension point.

func (*PreemptionToleration) SelectVictimsOnNode

func (pl *PreemptionToleration) SelectVictimsOnNode(
	ctx context.Context,
	state *framework.CycleState,
	preemptor *v1.Pod,
	nodeInfo *framework.NodeInfo,
	pdbs []*policy.PodDisruptionBudget) ([]*v1.Pod, int, *framework.Status)

SelectVictimsOnNode finds minimum set of pods on the given node that should be preempted in order to make enough room for "preemptor" to be scheduled. The algorithm is almost identical to DefaultPreemption plugin's one. The only difference is that it takes PreemptionToleration annotations in PriorityClass resources into account for selecting victim pods.