plugin

Documentation

Index

• Constants
• type AutoscaleEnforcer
  • func NewAutoscaleEnforcerPlugin(baseCtx context.Context, logger *zap.Logger, handle framework.Handle, ...) (_ *AutoscaleEnforcer, finalError error)
  • func (e *AutoscaleEnforcer) Filter(ctx context.Context, _state *framework.CycleState, pod *corev1.Pod, ...) (status *framework.Status)
  • func (e *AutoscaleEnforcer) Name() string
  • func (e *AutoscaleEnforcer) NormalizeScore(ctx context.Context, state *framework.CycleState, pod *corev1.Pod, ...) (status *framework.Status)
  • func (e *AutoscaleEnforcer) PostFilter(ctx context.Context, state *framework.CycleState, pod *corev1.Pod, ...) (_ *framework.PostFilterResult, status *framework.Status)
  • func (e *AutoscaleEnforcer) Reserve(ctx context.Context, _state *framework.CycleState, pod *corev1.Pod, ...) (status *framework.Status)
  • func (e *AutoscaleEnforcer) Score(ctx context.Context, _state *framework.CycleState, pod *corev1.Pod, ...) (_ int64, status *framework.Status)
  • func (e *AutoscaleEnforcer) ScoreExtensions() framework.ScoreExtensions
  • func (e *AutoscaleEnforcer) Unreserve(ctx context.Context, _state *framework.CycleState, pod *corev1.Pod, ...)
• type Config
  • func ReadConfig(path string) (*Config, error)
• type PluginState
  • func NewPluginState(config Config, vmClient vmclient.Interface, reg prometheus.Registerer, ...) *PluginState
  • func (s *PluginState) HandleMigrationEvent(logger *zap.Logger, kind reconcile.EventKind, ...) error
  • func (s *PluginState) HandleNodeEvent(logger *zap.Logger, kind reconcile.EventKind, node *corev1.Node) error
  • func (s *PluginState) HandlePodEvent(logger *zap.Logger, kind reconcile.EventKind, pod *corev1.Pod) (*reconcile.Result, error)
• type ScoringConfig

Constants

const (
	MaxHTTPBodySize  int64  = 1 << 10 // 1 KiB
	ContentTypeJSON  string = "application/json"
	ContentTypeError string = "text/plain"
)

const (
	MinPluginProtocolVersion api.PluginProtoVersion = api.PluginProtoV5_0
	MaxPluginProtocolVersion api.PluginProtoVersion = api.PluginProtoV5_0
)

const DefaultConfigPath = "/etc/scheduler-plugin-config/autoscale-enforcer-config.json"

const LabelPluginCreatedMigration = "autoscaling.neon.tech/created-by-scheduler"

LabelPluginCreatedMigration marks all VirtualMachineMigrations that are created automatically by the scheduler plugin.

const PluginName = "AutoscaleEnforcer"

Types

type AutoscaleEnforcer

type AutoscaleEnforcer struct {
	// contains filtered or unexported fields
}

AutoscaleEnforcer implements Kubernetes scheduling plugins to account for available autoscaling resources during scheduling.

For more info on k8s scheduling plugins, see: https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/

func NewAutoscaleEnforcerPlugin

func NewAutoscaleEnforcerPlugin(
	baseCtx context.Context,
	logger *zap.Logger,
	handle framework.Handle,
	config *Config,
) (_ *AutoscaleEnforcer, finalError error)

func (*AutoscaleEnforcer) Filter

func (e *AutoscaleEnforcer) Filter(
	ctx context.Context,
	_state *framework.CycleState,
	pod *corev1.Pod,
	nodeInfo *framework.NodeInfo,
) (status *framework.Status)

Filter gives our plugin a chance to signal that a pod shouldn't be put onto a particular node

Filter implements framework.FilterPlugin.

func (*AutoscaleEnforcer) Name

func (e *AutoscaleEnforcer) Name() string

Name returns the name of the AutoscaleEnforcer plugin

Name implements framework.Plugin.

func (*AutoscaleEnforcer) NormalizeScore

func (e *AutoscaleEnforcer) NormalizeScore(
	ctx context.Context,
	state *framework.CycleState,
	pod *corev1.Pod,
	scores framework.NodeScoreList,
) (status *framework.Status)

NormalizeScore weights scores uniformly in the range [minScore, trueScore], where minScore is framework.MinNodeScore + 1.

NormalizeScore implements framework.ScoreExtensions.

func (*AutoscaleEnforcer) PostFilter

func (e *AutoscaleEnforcer) PostFilter(
	ctx context.Context,
	state *framework.CycleState,
	pod *corev1.Pod,
	filteredNodeStatusMap framework.NodeToStatusMap,
) (_ *framework.PostFilterResult, status *framework.Status)

PostFilter is used by us for metrics on filter cycles that reject a Pod by filtering out all applicable nodes.

Quoting the docs for PostFilter:

> These plugins are called after Filter phase, but only when no feasible nodes were found for the > pod.

PostFilter implements framework.PostFilterPlugin.

func (*AutoscaleEnforcer) Reserve

func (e *AutoscaleEnforcer) Reserve(
	ctx context.Context,
	_state *framework.CycleState,
	pod *corev1.Pod,
	nodeName string,
) (status *framework.Status)

Reserve signals to our plugin that a particular pod will (probably) be bound to a node, giving us a chance to both (a) reserve the resources it needs within the node and (b) reject the pod if there aren't enough.

Reserve implements framework.ReservePlugin.

func (*AutoscaleEnforcer) Score

func (e *AutoscaleEnforcer) Score(
	ctx context.Context,
	_state *framework.CycleState,
	pod *corev1.Pod,
	nodeName string,
) (_ int64, status *framework.Status)

Score allows our plugin to express which nodes should be preferred for scheduling new pods onto

Even though this function is given (pod, node) pairs, our scoring is only really dependent on values of the node. However, we have special handling for when the pod no longer fits in the node (even though it might have during the Filter plugin) - we can't return a failure, because that would cause *all* scheduling of the pod to fail, so we instead return the minimum score.

The scores might not be consistent with each other, due to ongoing changes in the node. That's ok, because nothing relies on strict correctness here, and they should be approximately correct anyways.

Score implements framework.ScorePlugin.

func (*AutoscaleEnforcer) ScoreExtensions

func (e *AutoscaleEnforcer) ScoreExtensions() framework.ScoreExtensions

ScoreExtensions is required for framework.ScorePlugin, and can return nil if it's not used. However, we do use it, to randomize scores (when enabled).

func (*AutoscaleEnforcer) Unreserve

func (e *AutoscaleEnforcer) Unreserve(
	ctx context.Context,
	_state *framework.CycleState,
	pod *corev1.Pod,
	nodeName string,
)

Unreserve marks a pod as no longer on-track to being bound to a node, so we can release the resources we previously reserved for it.

Note: the documentation for ReservePlugin indicates that Unreserve both (a) must be idempotent and (b) may be called without a previous call to Reserve for the same pod.

Unreserve implements framework.ReservePlugin.

type Config

type Config struct {
	// Scoring defines our policies around how to weight where Pods should be scheduled.
	Scoring ScoringConfig `json:"scoring"`

	// Watermark is the fraction of total resources allocated above which we should be migrating VMs
	// away to reduce usage.
	Watermark float64 `json:"watermark"`

	// SchedulerName informs the scheduler of its name, so that it can identify pods that a previous
	// version handled.
	SchedulerName string `json:"schedulerName"`

	// ReconcileWorkers sets the number of parallel workers to use for the global reconcile queue.
	ReconcileWorkers int `json:"reconcileWorkers"`

	// LogSuccessiveFailuresThreshold is the threshold for number of failures in a row at which
	// we'll start logging that an object is failing to be reconciled.
	//
	// This is to help make it easier to go from metrics saying "N objects are failing" to actually
	// finding the relevant objects.
	LogSuccessiveFailuresThreshold int `json:"logSuccessiveFailuresThreshold"`

	// StartupEventHandlingTimeoutSeconds gives the maximum duration, in seconds, that we are
	// allowed to wait to finish handling all of the initial events generated by reading the cluster
	// state on startup.
	//
	// If event processing takes longer than this time, then plugin creation will fail, and the
	// scheduler pod will retry.
	StartupEventHandlingTimeoutSeconds int `json:"startupEventHandlingTimeoutSeconds"`

	// K8sCRUDTimeoutSeconds sets the timeout to use for creating, updating, or deleting singular
	// kubernetes objects.
	K8sCRUDTimeoutSeconds int `json:"k8sCRUDTimeoutSeconds"`

	// PatchRetryWaitSeconds sets the minimum duration, in seconds, that we must wait between
	// successive patch operations on a VirtualMachine object.
	PatchRetryWaitSeconds int `json:"patchRetryWaitSeconds"`

	// NodeMetricLabels gives additional labels to annotate node metrics with.
	// The map is keyed by the metric name, and gives the kubernetes label that should be used to
	// populate it.
	//
	// For example, we might use the following:
	//
	//   {
	//     "availability_zone": "topology.kubernetes.io/zone",
	//     "node_group": "eks.amazonaws.com/nodegroup"
	//   }
	NodeMetricLabels map[string]string `json:"nodeMetricLabels"`

	// IgnoredNamespaces, if provided, gives a list of namespaces that the plugin should completely
	// ignore, as if pods from those namespaces do not exist.
	//
	// This is specifically designed for our "overprovisioning" namespace, which creates paused pods
	// to trigger cluster-autoscaler.
	//
	// The only exception to this rule is during Filter method calls, where we do still count the
	// resources from such pods. The reason to do that is so that these overprovisioning pods can be
	// evicted, which will allow cluster-autoscaler to trigger scale-up.
	IgnoredNamespaces []string `json:"ignoredNamespaces"`
}

Config stores the global configuration for the scheduler plugin.

It is parsed from a JSON file in a separate ConfigMap.

func ReadConfig

func ReadConfig(path string) (*Config, error)

type PluginState

type PluginState struct {
	// contains filtered or unexported fields
}

PluginState stores the state of the scheduler plugin in its entirety

func NewPluginState

func NewPluginState(
	config Config,
	vmClient vmclient.Interface,
	reg prometheus.Registerer,
	podWatchStore *watch.Store[corev1.Pod],
	nodeWatchStore *watch.Store[corev1.Node],
) *PluginState

func (*PluginState) HandleMigrationEvent

func (s *PluginState) HandleMigrationEvent(
	logger *zap.Logger,
	kind reconcile.EventKind,
	vmm *vmv1.VirtualMachineMigration,
) error

func (*PluginState) HandleNodeEvent

func (s *PluginState) HandleNodeEvent(logger *zap.Logger, kind reconcile.EventKind, node *corev1.Node) error

func (*PluginState) HandlePodEvent

func (s *PluginState) HandlePodEvent(
	logger *zap.Logger,
	kind reconcile.EventKind,
	pod *corev1.Pod,
) (*reconcile.Result, error)

type ScoringConfig

type ScoringConfig struct {

	// MinUsageScore gives the ratio of the score at the minimum usage (i.e. 0) relative to the
	// score at the midpoint, which will have the maximum.
	//
	// This corresponds to y₀ in the desmos link above.
	MinUsageScore float64 `json:"minUsageScore"`
	// MaxUsageScore gives the ratio of the score at the maximum usage (i.e. full) relative to the
	// score at the midpoint, which will have the maximum.
	//
	// This corresponds to y₁ in the desmos link above.
	MaxUsageScore float64 `json:"maxUsageScore"`
	// ScorePeak gives the fraction at which the "target" or highest score should be, with the score
	// sloping down on either side towards MinUsageScore at 0 and MaxUsageScore at 1.
	//
	// This corresponds to xₚ in the desmos link.
	ScorePeak float64 `json:"scorePeak"`

	// Randomize, if true, will cause the scheduler to score a node with a random number in the
	// range [minScore + 1, trueScore], instead of the trueScore.
	Randomize bool
}