wao-scheduler

README

WAO Scheduler Version 2

A kube-scheduler with MinimizePower plugin and PodSpread plugin to schedule pods with WAO.

• Overview
• Getting Started
  • Installation
  • Deploy Pods with MinimizePower
    • Use Predicted Power Consumption
    • Use Predicted Response Time
      • Set Application Name
      • Set Weights
  • Deploy Pods with PodSpread
• Configuration
  • MinimizePowerArgs
• Development
  • Components
• Changelog
• Acknowledgements
• License

Overview

WAO Scheduler schedules pods with the WAO algorithm, which aims to minimize power consumption while keeping latency low, and optionally spread pods across nodes to keep availability high. This is done by the following scheduler plugins:

• MinimizePower: Score nodes based on predicted power consumption and response time.
• PodSpread: Keep availability high by spreading pods across nodes, works together with MinimizePower.

Getting Started

Installation

[!NOTE] Make sure you have wao-core and wao-metrics-adapter set up.

Install WAO Scheduler with default configuration.

kubectl apply -f https://github.com/waok8s/wao-scheduler/releases/download/v1.30.0/wao-scheduler.yaml

Wait for the scheduler to be ready.

kubectl wait pod $(kubectl get pods -n kube-system -l app=wao-scheduler -o jsonpath="{.items[0].metadata.name}") -n kube-system --for condition=Ready

Deploy Pods with MinimizePower

This plugin is enabled by default, so you only need to set spec.schedulerName.

[!WARNING] Note that this plugin requires that at least one container in the pod has requests.cpu or limits.cpu set, otherwise the pod will be rejected. To be exact, if requests.cpu is set, it will be used as the expected CPU usage of the pod, otherwise limits.cpu will be used, otherwise the pod will be rejected.

  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx
  spec:
+   schedulerName: wao-scheduler
    containers:
    - name: nginx
      image: nginx:1.14.2
      resources:
        requests:
          cpu: 100m
        limits:
          cpu: 200m

Use Predicted Power Consumption

The prerequisites are:

• wao-core and wao-metrics-adapter are set up.
• Power consumption prediction service is available.

So no additional configuration is needed in user's side.

Use Predicted Response Time

The prerequisites are:

• wao-core and wao-metrics-adapter are set up.
• Response time prediction service is available.
• Application name is set in Pod's labels (if your model requires it).
• Optionally, weights can be set in Pod's annotations to override the default values defined in the configuration.

Set Application Name

Application name is fetched from labels in the following order:

• wao.bitmedia.co.jp/app
• app.kubernetes.io/name
• app

In the following example, the application name is set to nginx1.

  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx
+   labels:
+     app.kubernetes.io/name: nginx1
+     app: nginx2

Set Weights

Weights may be different for each application, and can be set in the following format to override the default values defined in the configuration.

  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx
    labels:
      app: nginx
    annotations:
+     wao.bitmedia.co.jp/weight-power-consumption: "7"
+     wao.bitmedia.co.jp/weight-response-time: "3"

Deploy Pods with PodSpread

This plugin only effects pods controlled by Deployment (and ReplicaSet), and it needs to be enabled by setting wao.bitmedia.co.jp/podspread-rate annotation.

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment
    labels:
      app: nginx
+   annotations:
+     wao.bitmedia.co.jp/podspread-rate: "0.6"
  spec:
    replicas: 10
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
+       schedulerName: wao-scheduler
        containers:
        - name: nginx
          image: nginx:1.14.2

Then 60 percent of the pods will be spread across nodes, and the rest 40 percent will be scheduled normally (i.e. MinimizePower plugin will be used).

Configuration

Just like kube-scheduler, WAO Scheduler reads configuration from a file specified by --config flag.

Here is an example KubeSchedulerConfiguration with MinimizePower and PodSpread plugins enabled.

[!NOTE] It is recommended to use a higher weight for MinimizePower plugin, as the normalized score might have small difference between nodes, particularly when the cluster has many nodes.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: false
profiles:
  - schedulerName: wao-scheduler
    plugins:
      multiPoint:
        enabled:
        - name: MinimizePower
          weight: 20
        - name: PodSpread

MinimizePowerArgs

The following args can be set in pluginConfig to configure MinimizePower plugin.

    pluginConfig:
      - name: MinimizePower
        args:
          metricsCacheTTL: 15s
          predictorCacheTTL: 15m
          podUsageAssumption: 0.8
          cpuUsageFormat: Percent
          weightPowerConsumption: 1
          weightResponseTime: 1

Preset values can be found in config/base/cm.yaml, and default values can be found in pkg/plugins/minimizepower/types.go.

• metricsCacheTTL: The TTL of metrics cache. Too short TTL will cause frequent requests to metrics server.
• predictorCacheTTL: The TTL of predictor cache. Predictor always returns the same result for the same input, so it is safe to set a long TTL.
• podUsageAssumption: The rate of expected CPU usage for a pod that is binded to a node but not yet started. This is used to count the expected CPU usage when scheduling a set of pods (e.g. a Deployment). The scheduler will assume that a pending pod (that is binded to a node) will use requests.cpu * podUsageAssumption CPUs.
• cpuUsageFormat: The format of CPU usage send to predictor.
  • Raw: [0.0, NumLogicalCores]
  • Percent: [0.0, 100.0]
• weightPowerConsumption: The weight of power consumption in the score, must be >= 0.
• weightResponseTime: The weight of response time in the score, must be >= 0.

Development

This project is following Scheduling Framework.

Components

• pkg/plugins/minimizepower: MinimizePower plugin.
• pkg/plugins/podspread: PodSpread plugin.

Changelog

Versioning: we use the same major.minor as Kubernetes, and the patch is our own.

• 2024-05-17 v1.30.0
  • Support Kubernetes v1.30.
• 2024-05-07 v1.29.0
  • Support Kubernetes v1.29.
• 2024-03-29 v1.28.0
  • Support Kubernetes v1.28.
• 2024-03-04 v1.27.0
  • First release.
  • minimizepower Add the scheduler plugin.
  • podspread Add the scheduler plugin.

Acknowledgements

This work is supported by the New Energy and Industrial Technology Development Organization (NEDO) under its "Program to Develop and Promote the Commercialization of Energy Conservation Technologies to Realize a Decarbonized Society" (JPNP21005).

License

Copyright 2023 Bitmedia, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.