plugins

README

Scheduler Framework Plugins

Creating a new in-tree plugin

Read the docs to understand the different extension points within the scheduling framework.

TODO(#95156): flesh this out a bit more.

Adding plugin configuration parameters through KubeSchedulerConfiguration

You can give users the ability to configure parameters in scheduler plugins using KubeSchedulerConfiguration. This section covers how you can add arguments to existing in-tree plugins (example PR). Let's assume the plugin is called FooPlugin and we want to add an optional integer parameter named barParam.

Defining and registering the struct

First, we need to define a struct type named FooPluginArgs in pkg/scheduler/apis/config/types_pluginargs.go, which is the representation of the configuration parameters that is internal to the scheduler.

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type FooPluginArgs struct {
	// metav1 is k8s.io/apimachinery/pkg/apis/meta/v1
	metav1.TypeMeta
	BarParam int32
}

Note that we embed k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta to include API metadata for versioning and persistence. We add the +k8s:deepcopy-gen:interfaces comment to auto-generate a DeepCopy function for the struct.

Similarly, define FooPluginArgs in k8s.io/kube-scheduler/config/{version}/types_pluginargs.go, which is the versioned representation used in the kube-scheduler binary used for deserialization. This time, however, in order to allow implicit default values for arguments, the type of the struct's fields may be pointers; leaving a parameter unspecified will set the pointer field to its zero value (nil), which can be used to let the framework know that it must fill in the default value. BarParam is of type int32 and let's say we want a non-zero default value for it:

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type FooPluginArgs struct {
	metav1.TypeMeta `json:",inline"`
	BarParam *int32 `json:"barParam,omitempty"`
}

For each types_pluginargs.go addition, remember to register the type in the corresponding register.go, which will allow the scheduler to recognize KubeSchedulerConfiguration values at parse-time.

Setting defaults

When a KubeSchedulerConfiguration object is parsed (happens in cmd/kube-scheduler/app/options/options.go), the scheduler will convert from the versioned type to the internal type, filling in the unspecified fields with defaults. Speaking of defaults, define SetDefaults_FooPluginArgs in pkg/scheduler/apis/config/v1beta1/defaults.go as follows:

// v1beta1 refers to k8s.io/kube-scheduler/config/v1beta1.
func SetDefaults_FooPluginArgs(obj *v1beta1.FooPluginArgs) {
	if obj.BarParam == nil {
		obj.BarParam = pointer.Int32Ptr(42)
	}
}

Validating configuration at runtime

Next, we need to define validators to make sure the user's configuration and your default values are valid. To do this, add something like this in pkg/scheduler/apis/config/validation/validation_pluginargs.go:

// From here on, FooPluginArgs refers to the type defined in pkg/scheduler
// definition, not the kube-scheduler definition. We're dealing with
// post-default values.
func ValidateFooPluginArgs(args config.FooPluginArgs) error {
	if args.BarParam < 0 && args.BarParam > 100 {
		return fmt.Errorf("must be in the range [0, 100]")
	}
	return nil
}

Code generation

We have defined everything necessary to run code generation now. Remember to commit all your changes (not sure why this is needed) and do a make clean first. Then:

$ cd $GOPATH/src/k8s.io/kubernetes
$ git add -A && git commit
$ make clean
$ ./hack/update-codegen.sh
$ make generated_files

This should automatically generate code to deep copy objects, convert between different struct types, convert pointer types to raw types, and set defaults.

Testing

After code generation, go back and write tests for all of the changes you made in the previous section:

• pkg/scheduler/apis/config/v1beta1/defaults_test.go to unit test the defaults.
• pkg/scheduler/apis/config/validation/validation_pluginargs_test.go to unit test the validator.
• pkg/scheduler/apis/config/scheme/scheme_test.go to test the whole pipeline using a KubeSchedulerConfiguration definition.

Receiving the arguments in the plugin

We can now finally receive FooPluginArgs in the plugin code. To do this, modify the plugin's New method signature like so:

func New(fpArgs runtime.Object, fh framework.FrameworkHandle) (framework.Plugin, error) {
	// config.FooPluginArgs refers to the pkg/scheduler struct type definition.
	args, ok := fpArgs.(*config.FooPluginArgs)
	if !ok {
		return nil, fmt.Errorf("got args of type %T, want *FooPluginArgs", fpArgs)
	}
	if err := validation.ValidateFooPluginArgs(*args); err != nil {
		return nil, err
	}
	// Use args.BarParam as you like.
}

Documentation

Index

• Constants
• func NewInTreeRegistry() runtime.Registry
• type ConfigProducerArgs
• type LegacyRegistry
  • func NewLegacyRegistry() *LegacyRegistry
  • func (lr *LegacyRegistry) AppendPredicateConfigs(keys sets.String, args *ConfigProducerArgs, plugins config.Plugins, ...) (config.Plugins, []config.PluginConfig, error)
  • func (lr *LegacyRegistry) AppendPriorityConfigs(keys map[string]int64, args *ConfigProducerArgs, plugins config.Plugins, ...) (config.Plugins, []config.PluginConfig, error)
  • func (lr *LegacyRegistry) ProcessPredicatePolicy(policy config.PredicatePolicy, pluginArgs *ConfigProducerArgs) string
  • func (lr *LegacyRegistry) ProcessPriorityPolicy(policy config.PriorityPolicy, configProducerArgs *ConfigProducerArgs) string

Constants

const (
	// EqualPriority defines the name of prioritizer function that gives an equal weight of one to all nodes.
	EqualPriority = "EqualPriority"
	// MostRequestedPriority defines the name of prioritizer function that gives used nodes higher priority.
	MostRequestedPriority = "MostRequestedPriority"
	// RequestedToCapacityRatioPriority defines the name of RequestedToCapacityRatioPriority.
	RequestedToCapacityRatioPriority = "RequestedToCapacityRatioPriority"
	// SelectorSpreadPriority defines the name of prioritizer function that spreads pods by minimizing
	// the number of pods (belonging to the same service or replication controller) on the same node.
	SelectorSpreadPriority = "SelectorSpreadPriority"
	// ServiceSpreadingPriority is largely replaced by "SelectorSpreadPriority".
	ServiceSpreadingPriority = "ServiceSpreadingPriority"
	// InterPodAffinityPriority defines the name of prioritizer function that decides which pods should or
	// should not be placed in the same topological domain as some other pods.
	InterPodAffinityPriority = "InterPodAffinityPriority"
	// LeastRequestedPriority defines the name of prioritizer function that prioritize nodes by least
	// requested utilization.
	LeastRequestedPriority = "LeastRequestedPriority"
	// BalancedResourceAllocation defines the name of prioritizer function that prioritizes nodes
	// to help achieve balanced resource usage.
	BalancedResourceAllocation = "BalancedResourceAllocation"
	// NodePreferAvoidPodsPriority defines the name of prioritizer function that priorities nodes according to
	// the node annotation "scheduler.alpha.kubernetes.io/preferAvoidPods".
	NodePreferAvoidPodsPriority = "NodePreferAvoidPodsPriority"
	// NodeAffinityPriority defines the name of prioritizer function that prioritizes nodes which have labels
	// matching NodeAffinity.
	NodeAffinityPriority = "NodeAffinityPriority"
	// TaintTolerationPriority defines the name of prioritizer function that prioritizes nodes that marked
	// with taint which pod can tolerate.
	TaintTolerationPriority = "TaintTolerationPriority"
	// ImageLocalityPriority defines the name of prioritizer function that prioritizes nodes that have images
	// requested by the pod present.
	ImageLocalityPriority = "ImageLocalityPriority"
	// EvenPodsSpreadPriority defines the name of prioritizer function that prioritizes nodes
	// which have pods and labels matching the incoming pod's topologySpreadConstraints.
	EvenPodsSpreadPriority = "EvenPodsSpreadPriority"
)

const (
	// MatchInterPodAffinityPred defines the name of predicate MatchInterPodAffinity.
	MatchInterPodAffinityPred = "MatchInterPodAffinity"
	// CheckVolumeBindingPred defines the name of predicate CheckVolumeBinding.
	CheckVolumeBindingPred = "CheckVolumeBinding"
	// GeneralPred defines the name of predicate GeneralPredicates.
	GeneralPred = "GeneralPredicates"
	// HostNamePred defines the name of predicate HostName.
	HostNamePred = "HostName"
	// PodFitsHostPortsPred defines the name of predicate PodFitsHostPorts.
	PodFitsHostPortsPred = "PodFitsHostPorts"
	// MatchNodeSelectorPred defines the name of predicate MatchNodeSelector.
	MatchNodeSelectorPred = "MatchNodeSelector"
	// PodFitsResourcesPred defines the name of predicate PodFitsResources.
	PodFitsResourcesPred = "PodFitsResources"
	// NoDiskConflictPred defines the name of predicate NoDiskConflict.
	NoDiskConflictPred = "NoDiskConflict"
	// PodToleratesNodeTaintsPred defines the name of predicate PodToleratesNodeTaints.
	PodToleratesNodeTaintsPred = "PodToleratesNodeTaints"
	// CheckNodeUnschedulablePred defines the name of predicate CheckNodeUnschedulablePredicate.
	CheckNodeUnschedulablePred = "CheckNodeUnschedulable"
	// CheckNodeLabelPresencePred defines the name of predicate CheckNodeLabelPresence.
	CheckNodeLabelPresencePred = "CheckNodeLabelPresence"
	// CheckServiceAffinityPred defines the name of predicate checkServiceAffinity.
	CheckServiceAffinityPred = "CheckServiceAffinity"
	// MaxEBSVolumeCountPred defines the name of predicate MaxEBSVolumeCount.
	// DEPRECATED
	// All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred.
	MaxEBSVolumeCountPred = "MaxEBSVolumeCount"
	// MaxGCEPDVolumeCountPred defines the name of predicate MaxGCEPDVolumeCount.
	// DEPRECATED
	// All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred.
	MaxGCEPDVolumeCountPred = "MaxGCEPDVolumeCount"
	// MaxAzureDiskVolumeCountPred defines the name of predicate MaxAzureDiskVolumeCount.
	// DEPRECATED
	// All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred.
	MaxAzureDiskVolumeCountPred = "MaxAzureDiskVolumeCount"
	// MaxCinderVolumeCountPred defines the name of predicate MaxCinderDiskVolumeCount.
	// DEPRECATED
	// All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred.
	MaxCinderVolumeCountPred = "MaxCinderVolumeCount"
	// MaxCSIVolumeCountPred defines the predicate that decides how many CSI volumes should be attached.
	MaxCSIVolumeCountPred = "MaxCSIVolumeCountPred"
	// NoVolumeZoneConflictPred defines the name of predicate NoVolumeZoneConflict.
	NoVolumeZoneConflictPred = "NoVolumeZoneConflict"
	// EvenPodsSpreadPred defines the name of predicate EvenPodsSpread.
	EvenPodsSpreadPred = "EvenPodsSpread"
)

Functions

func NewInTreeRegistry

func NewInTreeRegistry() runtime.Registry

NewInTreeRegistry builds the registry with all the in-tree plugins. A scheduler that runs out of tree plugins can register additional plugins through the WithFrameworkOutOfTreeRegistry option.

Types

type ConfigProducerArgs

type ConfigProducerArgs struct {
	// Weight used for priority functions.
	Weight int32
	// NodeLabelArgs is the args for the NodeLabel plugin.
	NodeLabelArgs *config.NodeLabelArgs
	// RequestedToCapacityRatioArgs is the args for the RequestedToCapacityRatio plugin.
	RequestedToCapacityRatioArgs *config.RequestedToCapacityRatioArgs
	// ServiceAffinityArgs is the args for the ServiceAffinity plugin.
	ServiceAffinityArgs *config.ServiceAffinityArgs
	// NodeResourcesFitArgs is the args for the NodeResources fit filter.
	NodeResourcesFitArgs *config.NodeResourcesFitArgs
	// InterPodAffinityArgs is the args for InterPodAffinity plugin
	InterPodAffinityArgs *config.InterPodAffinityArgs
}

ConfigProducerArgs contains arguments that are passed to the producer. As we add more predicates/priorities to framework plugins mappings, more arguments may be added here.

type LegacyRegistry

type LegacyRegistry struct {

	// predicates and priorities that will be used if either was set to nil in a
	// given v1.Policy configuration.
	DefaultPredicates sets.String
	DefaultPriorities map[string]int64
	// contains filtered or unexported fields
}

LegacyRegistry is used to store current state of registered predicates and priorities.

func NewLegacyRegistry

func NewLegacyRegistry() *LegacyRegistry

NewLegacyRegistry returns a legacy algorithm registry of predicates and priorities.

func (*LegacyRegistry) AppendPredicateConfigs

func (lr *LegacyRegistry) AppendPredicateConfigs(keys sets.String, args *ConfigProducerArgs, plugins config.Plugins, pluginConfig []config.PluginConfig) (config.Plugins, []config.PluginConfig, error)

AppendPredicateConfigs returns predicates configuration that will run as framework plugins. Note that the framework executes plugins according to their order in the Plugins list, and so predicates run as plugins are added to the Plugins list according to the order specified in predicateOrdering.

func (*LegacyRegistry) AppendPriorityConfigs

func (lr *LegacyRegistry) AppendPriorityConfigs(keys map[string]int64, args *ConfigProducerArgs, plugins config.Plugins, pluginConfig []config.PluginConfig) (config.Plugins, []config.PluginConfig, error)

AppendPriorityConfigs returns priorities configuration that will run as framework plugins.

func (*LegacyRegistry) ProcessPredicatePolicy

func (lr *LegacyRegistry) ProcessPredicatePolicy(policy config.PredicatePolicy, pluginArgs *ConfigProducerArgs) string

ProcessPredicatePolicy given a PredicatePolicy, return the plugin name implementing the predicate and update the ConfigProducerArgs if necessary.

func (*LegacyRegistry) ProcessPriorityPolicy

func (lr *LegacyRegistry) ProcessPriorityPolicy(policy config.PriorityPolicy, configProducerArgs *ConfigProducerArgs) string

ProcessPriorityPolicy given a PriorityPolicy, return the plugin name implementing the priority and update the ConfigProducerArgs if necessary.