scheduler

README

qemu-scheduler

Scheduling refers to making sure that VM(QEMU) are matched to Proxmox Nodes.

How qemu-scheduler select proxmox node to run qemu

Basic flow of the node selection process is filter => score => select one node which has highest score

Filter Plugins

Filter plugins filter the node based on nodename, overcommit ratio etc. So that we can avoid to run qemus on not desired Proxmox nodes.

• NodeName plugin (pass the node matching specified node name)
• CPUOvercommit plugin (pass the node that has enough cpu against running vm)
• MemoryOvercommit plugin (pass the node that has enough memory against running vm)
• NodeRegex plugin (pass the node matching specified regex)

regex plugin

Regex plugin is a one of the default Filter Plugin of qemu-scheduler. You can specify node name as regex format.

key: node.qemu-scheduler/regex
value(example): node[0-9]+

Score Plugins

Score plugins score the nodes based on resource etc. So that we can run qemus on the most appropriate Proxmox node.

• NodeResource plugin (nodes with more resources have higher scores)
• Random plugin (diabled by default. just a reference implementation of score plugin)

How to specify vmid

qemu-scheduler reads context and find key registerd to scheduler. If the context has any value of the registerd key, qemu-scheduler uses the plugin that matchies the key.

• Range plugin (select minimum availabe vmid from the specified id range)
• VMIDRegex plugin (select minimum availabe vmid matching specified regex)

Range Plugin

You can specify vmid range with (start id)-(end id) format.

key: vmid.qemu-scheduler/range
value(example): 100-150

Regex Plugin

key: vmid.qemu-scheduler/regex
value(example): (12[0-9]|130)

How qemu-scheduler works with CAPPX

CAPPX passes all the annotation (of ProxmoxMachine) key-values to scheduler's context. So if you will use Range Plugin for your ProxmoxMachine, your manifest must look like following.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ProxmoxMachine
metadata:
    name: sample-machine
    annotations:
        vmid.qemu-scheduler/range: 100-150 # this means your vmid will be chosen from the range of 100 to 150.

Also, you can specifies these annotations via MachineDeployment since Cluster API propagates some metadatas (ref: metadata-propagation).

For example, your MachineDeployment may look like following.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
    caution: "# do not use here, because this annotation won't be propagated to your ProxmoxMachine"
  name: sample-machine-deployment
spec:
  template:
    metadata:
      annotations:
        node.qemu-scheduler/regex: node[0-9]+ # this annotation will be propagated to your ProxmoxMachine via MachineSet

How to configure (or disable/enable) specific Plugins

By default, all the plugins are enabled. You can disable specific plugins via plugin-config. for CAPPX, check example ConfigMap here

# example plugin-config.yaml

# plugin type name (scores, filters, vmids)
filters:
  CPUOvercommit:
    enable: false # disable
  MemoryOvercommit:
    enable: true   # enable (can be omitted)
vmids:
  Regex:
    enable: false # disable

Documentation

Index

• Variables
• type CancelFunc
• type Manager
  • func NewManager(params SchedulerParams) (*Manager, error)
  • func (m *Manager) GetOrCreateScheduler(client *proxmox.Service) *Scheduler
  • func (m *Manager) NewScheduler(client *proxmox.Service, opts ...SchedulerOption) *Scheduler
• type Scheduler
  • func (s *Scheduler) CreateQEMU(ctx context.Context, config *api.VirtualMachineCreateOptions) (framework.SchedulerResult, error)
  • func (s *Scheduler) IsRunning() bool
  • func (s *Scheduler) Run()
  • func (s *Scheduler) RunAsync()
  • func (s *Scheduler) RunFilterPlugins(ctx context.Context, state *framework.CycleState, ...) ([]*api.Node, error)
  • func (s *Scheduler) RunScorePlugins(ctx context.Context, state *framework.CycleState, ...) (framework.NodeScoreList, *framework.Status)
  • func (s *Scheduler) RunVMIDPlugins(ctx context.Context, state *framework.CycleState, ...) (int, error)
  • func (s *Scheduler) ScheduleOne(ctx context.Context)
  • func (s *Scheduler) SelectNode(ctx context.Context, config api.VirtualMachineCreateOptions) (string, error)
  • func (s *Scheduler) SelectVMID(ctx context.Context, config api.VirtualMachineCreateOptions) (int, error)
  • func (s *Scheduler) Stop()
  • func (s *Scheduler) WaitStatus(ctx context.Context, config *api.VirtualMachineCreateOptions) (framework.CycleState, error)
  • func (s *Scheduler) WithCancel() (*Scheduler, CancelFunc)
• type SchedulerOption
  • func WithTimeout(timeout time.Duration) SchedulerOption
• type SchedulerParams
  • func (p *SchedulerParams) PluginConfigs() plugins.PluginConfigs

Variables

var (
	// ErrNoNodesAvailable is used to describe the error that no nodes available to schedule qemus.
	ErrNoNodesAvailable = fmt.Errorf("no nodes available to schedule qemus")

	// ErrNoVMIDAvailable is used to describe the error that no vmid available to schedule qemus.
	ErrNoVMIDAvailable = fmt.Errorf("no vmid available to schedule qemus")
)

Types

type CancelFunc

type CancelFunc func()

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

manager manages schedulers

func NewManager

func NewManager(params SchedulerParams) (*Manager, error)

return manager with initialized scheduler-table

func (*Manager) GetOrCreateScheduler

func (m *Manager) GetOrCreateScheduler(client *proxmox.Service) *Scheduler

return new/existing scheduler

func (*Manager) NewScheduler

func (m *Manager) NewScheduler(client *proxmox.Service, opts ...SchedulerOption) *Scheduler

return new scheduler. usually better to use GetOrCreateScheduler instead.

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

func (*Scheduler) CreateQEMU

func (s *Scheduler) CreateQEMU(ctx context.Context, config *api.VirtualMachineCreateOptions) (framework.SchedulerResult, error)

create new qemu with given spec and context

func (*Scheduler) IsRunning

func (s *Scheduler) IsRunning() bool

func (*Scheduler) Run

func (s *Scheduler) Run()

run scheduler and ensure only one process is running

func (*Scheduler) RunAsync

func (s *Scheduler) RunAsync()

run scheduelr in parallel

func (*Scheduler) RunFilterPlugins

func (s *Scheduler) RunFilterPlugins(ctx context.Context, state *framework.CycleState, config api.VirtualMachineCreateOptions, nodes []*api.Node) ([]*api.Node, error)

func (*Scheduler) RunScorePlugins

func (s *Scheduler) RunScorePlugins(ctx context.Context, state *framework.CycleState, config api.VirtualMachineCreateOptions, nodes []*api.Node) (framework.NodeScoreList, *framework.Status)

func (*Scheduler) RunVMIDPlugins

func (s *Scheduler) RunVMIDPlugins(ctx context.Context, state *framework.CycleState, config api.VirtualMachineCreateOptions, nextid int, usedID map[int]bool) (int, error)

func (*Scheduler) ScheduleOne

func (s *Scheduler) ScheduleOne(ctx context.Context)

retrieve one qemuSpec from queue and try to create new qemu according to the qemuSpec

func (*Scheduler) SelectNode

func (s *Scheduler) SelectNode(ctx context.Context, config api.VirtualMachineCreateOptions) (string, error)

func (*Scheduler) SelectVMID

func (s *Scheduler) SelectVMID(ctx context.Context, config api.VirtualMachineCreateOptions) (int, error)

func (*Scheduler) Stop

func (s *Scheduler) Stop()

stop scheduler

func (*Scheduler) WaitStatus

func (s *Scheduler) WaitStatus(ctx context.Context, config *api.VirtualMachineCreateOptions) (framework.CycleState, error)

wait until CycleState is put into channel and then return it

func (*Scheduler) WithCancel

func (s *Scheduler) WithCancel() (*Scheduler, CancelFunc)

type SchedulerOption

type SchedulerOption func(s *Scheduler)

func WithTimeout

func WithTimeout(timeout time.Duration) SchedulerOption

set timeout to scheduler

type SchedulerParams

type SchedulerParams struct {
	Logger logr.Logger

	// file path for pluginConfig
	PluginConfigFile string
	// contains filtered or unexported fields
}

func (*SchedulerParams) PluginConfigs

func (p *SchedulerParams) PluginConfigs() plugins.PluginConfigs