k8shorizmetrics

README

k8shorizmetrics

k8shorizmetrics is a library that provides the internal workings of the Kubernetes Horizontal Pod Autoscaler (HPA) wrapped up in a simple API. The project allows querying metrics just as the HPA does, and also running the calculations to work out the target replica count that the HPA does.

Install

go get -u github.com/jthomperoo/k8shorizmetrics/v3@v3.0.0

Features

• Simple API, based directly on the code from the HPA, but detangled for ease of use.
• Dependent only on versioned and public Kubernetes Golang modules, allows easy install without replace directives.
• Splits the HPA into two parts, metric gathering and evaluation, only use what you need.
• Allows insights into how the HPA makes decisions.
• Supports scaling to and from 0.

Quick Start

The following is a simple program that can run inside a Kubernetes cluster that gets the CPU resource metrics for pods with the label run: php-apache.

package main

import (
	"log"
	"time"

	"github.com/jthomperoo/k8shorizmetrics"
	"github.com/jthomperoo/k8shorizmetrics/metricsclient"
	"github.com/jthomperoo/k8shorizmetrics/podsclient"
	"k8s.io/api/autoscaling/v2"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/labels"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
)

func main() {
	// Kubernetes API setup
	clusterConfig, _ := rest.InClusterConfig()
	clientset, _ := kubernetes.NewForConfig(clusterConfig)
	// Metrics and pods clients setup
	metricsclient := metricsclient.NewClient(clusterConfig, clientset.Discovery())
	podsclient := &podsclient.OnDemandPodLister{Clientset: clientset}
	// HPA configuration options
	cpuInitializationPeriod := time.Duration(300) * time.Second
	initialReadinessDelay := time.Duration(30) * time.Second

	// Setup gatherer
	gather := k8shorizmetrics.NewGatherer(metricsclient, podsclient, cpuInitializationPeriod, initialReadinessDelay)

	// Target resource values
	namespace := "default"
	podSelector := labels.SelectorFromSet(labels.Set{
		"run": "php-apache",
	})

	// Metric spec to gather, CPU resource utilization
	spec := v2.MetricSpec{
		Type: v2.ResourceMetricSourceType,
		Resource: &v2.ResourceMetricSource{
			Name: corev1.ResourceCPU,
			Target: v2.MetricTarget{
				Type: v2.UtilizationMetricType,
			},
		},
	}

	metric, _ := gather.GatherSingleMetric(spec, namespace, podSelector)

	for pod, podmetric := range metric.Resource.PodMetricsInfo {
		actualCPU := podmetric.Value
		requestedCPU := metric.Resource.Requests[pod]
		log.Printf("Pod: %s, CPU usage: %dm (%0.2f%% of requested)\n", pod, actualCPU, float64(actualCPU)/float64(requestedCPU)*100.0)
	}
}

Documentation

See the Go doc.

Migration

This section explains how to migrate between versions of the library.

From v1 to v2

There are two changes you need to make to migrate from v1 to v2:

1. Switch from using k8s.io/api/autoscaling/v2beta2 to k8s.io/api/autoscaling/v2.
2. Switch from using github.com/jthomperoo/k8shorizmetrics to github.com/jthomperoo/k8shorizmetrics/v2.

From v2 to v3

The breaking changes introduced by v3 are:

• Gather now returns the GathererMultiMetricError error type if any of the metrics fail to gather. This error is returned for partial errors, meaning some metrics gathered successfully and others did not. If this partial error occurs the GathererMultiMetricError error will have the Partial property set to true. This can be checked for using errors.As.
• Evaluate now returns the EvaluatorMultiMetricError error type if any of the metrics fail to evaluate. This error is returned for partial errors, meaning some metrics evaluted successfully and others did not. If this partial error occurs the EvaluatorMultiMetricError error will have the Partial property set to true. This can be checked for using errors.As.

To update to v3 you will need to update all references in your code that refer to github.com/jthomperoo/k8shorizmetrics/v2 to use github.com/jthomperoo/k8shorizmetrics/v3.

If you want the behaviour to stay the same and to swallow partial errors you can use code like this:

metrics, err := gather.Gather(specs, namespace, podMatchSelector)
if err != nil {
	gatherErr := &k8shorizmetrics.GathererMultiMetricError{}
	if !errors.As(err, &gatherErr) {
		log.Fatal(err)
	}

	if !gatherErr.Partial {
		log.Fatal(err)
	}

	// Not a partial error, just continue as normal
}

You can use similar code for the Evaluate method of the Evaluater.

Examples

See the examples directory for some examples, cpuprint is a good start.

Developing and Contributing

See the contribution guidelines and code of conduct.

Documentation

Overview

Package k8shorizmetrics provides a simplified interface for gathering metrics and calculating replicas in the same way that the Horizontal Pod Autoscaler (HPA) does. This is split into two parts, gathering metrics, and evaluating metrics (calculating replicas). You can use these parts separately, or together to create a full evaluation process in the same way the HPA does.

Index

• type Evaluator
  • func NewEvaluator(tolerance float64) *Evaluator
  • func (e *Evaluator) Evaluate(gatheredMetrics []*metrics.Metric, currentReplicas int32) (int32, error)
  • func (e *Evaluator) EvaluateSingleMetric(gatheredMetric *metrics.Metric, currentReplicas int32) (int32, error)
  • func (e *Evaluator) EvaluateSingleMetricWithOptions(gatheredMetric *metrics.Metric, currentReplicas int32, tolerance float64) (int32, error)
  • func (e *Evaluator) EvaluateWithOptions(gatheredMetrics []*metrics.Metric, currentReplicas int32, tolerance float64) (int32, error)
• type EvaluatorMultiMetricError
  • func (e *EvaluatorMultiMetricError) Error() string
• type ExternalEvaluater
• type ExternalGatherer
• type Gatherer
  • func NewGatherer(metricsclient metricsclient.Client, podlister corelisters.PodLister, ...) *Gatherer
  • func (c *Gatherer) Gather(specs []autoscalingv2.MetricSpec, namespace string, ...) ([]*metrics.Metric, error)
  • func (c *Gatherer) GatherSingleMetric(spec autoscalingv2.MetricSpec, namespace string, podSelector labels.Selector) (*metrics.Metric, error)
  • func (c *Gatherer) GatherSingleMetricWithOptions(spec autoscalingv2.MetricSpec, namespace string, podSelector labels.Selector, ...) (*metrics.Metric, error)
  • func (c *Gatherer) GatherWithOptions(specs []autoscalingv2.MetricSpec, namespace string, ...) ([]*metrics.Metric, error)
• type GathererMultiMetricError
  • func (e *GathererMultiMetricError) Error() string
• type ObjectEvaluater
• type ObjectGatherer
• type PodsEvaluater
• type PodsGatherer
• type ResourceEvaluater
• type ResourceGatherer

Types

type Evaluator

type Evaluator struct {
	External  ExternalEvaluater
	Object    ObjectEvaluater
	Pods      PodsEvaluater
	Resource  ResourceEvaluater
	Tolerance float64
}

Evaluator provides functionality for deciding how many replicas a resource should have based on provided metrics.

func NewEvaluator

func NewEvaluator(tolerance float64) *Evaluator

NewEvaluator sets up an evaluate that can process external, object, pod and resource metrics

func (*Evaluator) Evaluate

func (e *Evaluator) Evaluate(gatheredMetrics []*metrics.Metric, currentReplicas int32) (int32, error)

Evaluate returns the target replica count for an array of multiple metrics If an error occurs evaluating any metric this will return a EvaluatorMultiMetricError. If a partial error occurs, meaning some metrics were evaluated successfully and others failed, the 'Partial' property of this error will be set to true.

func (*Evaluator) EvaluateSingleMetric

func (e *Evaluator) EvaluateSingleMetric(gatheredMetric *metrics.Metric, currentReplicas int32) (int32, error)

EvaluateSingleMetric returns the target replica count for a single metrics

func (*Evaluator) EvaluateSingleMetricWithOptions

func (e *Evaluator) EvaluateSingleMetricWithOptions(gatheredMetric *metrics.Metric, currentReplicas int32,
	tolerance float64) (int32, error)

EvaluateSingleMetricWithOptions returns the target replica count for a single metrics with provided options

func (*Evaluator) EvaluateWithOptions

func (e *Evaluator) EvaluateWithOptions(gatheredMetrics []*metrics.Metric, currentReplicas int32,
	tolerance float64) (int32, error)

EvaluateWithOptions returns the target replica count for an array of multiple metrics with provided options If an error occurs evaluating any metric this will return a EvaluatorMultiMetricError. If a partial error occurs, meaning some metrics were evaluated successfully and others failed, the 'Partial' property of this error will be set to true.

type EvaluatorMultiMetricError

type EvaluatorMultiMetricError struct {
	Partial bool
	Errors  []error
}

EvaluatorMultiMetricError occurs when evaluating multiple metrics, if any metric fails to be evaluated this error will be returned which contains all of the individual errors in the 'Errors' slice, if some metrics were evaluated successfully the error will have the 'Partial' property set to true.

func (*EvaluatorMultiMetricError) Error

func (e *EvaluatorMultiMetricError) Error() string

type ExternalEvaluater

type ExternalEvaluater interface {
	Evaluate(currentReplicas int32, gatheredMetric *metrics.Metric, tolerance float64) (int32, error)
}

ExternalEvaluater produces a replica count based on an external metric provided

type ExternalGatherer

type ExternalGatherer interface {
	Gather(metricName, namespace string, metricSelector *metav1.LabelSelector, podSelector labels.Selector) (*externalmetrics.Metric, error)
	GatherPerPod(metricName, namespace string, metricSelector *metav1.LabelSelector) (*externalmetrics.Metric, error)
}

ExternalGatherer allows retrieval of external metrics.

type Gatherer

type Gatherer struct {
	Resource                      ResourceGatherer
	Pods                          PodsGatherer
	Object                        ObjectGatherer
	External                      ExternalGatherer
	ScaleClient                   k8sscale.ScalesGetter
	CPUInitializationPeriod       time.Duration
	DelayOfInitialReadinessStatus time.Duration
}

Gatherer provides functionality for retrieving metrics on supplied metric specs.

func NewGatherer

func NewGatherer(
	metricsclient metricsclient.Client,
	podlister corelisters.PodLister,
	cpuInitializationPeriod time.Duration,
	delayOfInitialReadinessStatus time.Duration) *Gatherer

NewGatherer sets up a new Metric Gatherer

func (*Gatherer) Gather

func (c *Gatherer) Gather(specs []autoscalingv2.MetricSpec, namespace string, podSelector labels.Selector) ([]*metrics.Metric, error)

Gather returns all of the metrics gathered based on the metric specs provided. If an error occurs gathering any metric this will return a GatherMultiMetricError. If a partial error occurs, meaning some metrics were gathered successfully and others failed, the 'Partial' property of this error will be set to true.

func (*Gatherer) GatherSingleMetric

func (c *Gatherer) GatherSingleMetric(spec autoscalingv2.MetricSpec, namespace string, podSelector labels.Selector) (*metrics.Metric, error)

GatherSingleMetric returns the metric gathered based on a single metric spec.

func (*Gatherer) GatherSingleMetricWithOptions

func (c *Gatherer) GatherSingleMetricWithOptions(spec autoscalingv2.MetricSpec, namespace string, podSelector labels.Selector,
	cpuInitializationPeriod time.Duration, delayOfInitialReadinessStatus time.Duration) (*metrics.Metric, error)

GatherSingleMetricWithOptions returns the metric gathered based on a single metric spec with options.

func (*Gatherer) GatherWithOptions

func (c *Gatherer) GatherWithOptions(specs []autoscalingv2.MetricSpec, namespace string, podSelector labels.Selector,
	cpuInitializationPeriod time.Duration, delayOfInitialReadinessStatus time.Duration) ([]*metrics.Metric, error)

GatherWithOptions returns all of the metrics gathered based on the metric specs provided with options. If an error occurs gathering any metric this will return a GatherMultiMetricError. If a partial error occurs, meaning some metrics were gathered successfully and others failed, the 'Partial' property of this error will be set to true.

type GathererMultiMetricError

type GathererMultiMetricError struct {
	Partial bool
	Errors  []error
}

GathererMultiMetricError occurs when gathering multiple metrics, if any metric fails to be gathered this error will be returned which contains all of the individual errors in the 'Errors' slice, if some metrics were gathered successfully the error will have the 'Partial' property set to true.

func (*GathererMultiMetricError) Error

func (e *GathererMultiMetricError) Error() string

type ObjectEvaluater

type ObjectEvaluater interface {
	Evaluate(currentReplicas int32, gatheredMetric *metrics.Metric, tolerance float64) (int32, error)
}

ObjectEvaluater produces a replica count based on an object metric provided

type ObjectGatherer

type ObjectGatherer interface {
	Gather(metricName string, namespace string, objectRef *autoscalingv2.CrossVersionObjectReference, podSelector labels.Selector, metricSelector labels.Selector) (*objectmetrics.Metric, error)
	GatherPerPod(metricName string, namespace string, objectRef *autoscalingv2.CrossVersionObjectReference, metricSelector labels.Selector) (*objectmetrics.Metric, error)
}

ObjectGatherer allows retrieval of object metrics.

type PodsEvaluater

type PodsEvaluater interface {
	Evaluate(currentReplicas int32, gatheredMetric *metrics.Metric) int32
}

PodsEvaluater produces a replica count based on a pods metric provided

type PodsGatherer

type PodsGatherer interface {
	Gather(metricName string, namespace string, podSelector labels.Selector, metricSelector labels.Selector) (*podsmetrics.Metric, error)
}

PodsGatherer allows retrieval of pods metrics.

type ResourceEvaluater

type ResourceEvaluater interface {
	Evaluate(currentReplicas int32, gatheredMetric *metrics.Metric, tolerance float64) (int32, error)
}

ResourceEvaluater produces an evaluation based on a resource metric provided

type ResourceGatherer

type ResourceGatherer interface {
	Gather(resourceName corev1.ResourceName, namespace string, podSelector labels.Selector,
		cpuInitializationPeriod time.Duration, delayOfInitialReadinessStatus time.Duration) (*resourcemetrics.Metric, error)
	GatherRaw(resourceName corev1.ResourceName, namespace string, podSelector labels.Selector,
		cpuInitializationPeriod time.Duration, delayOfInitialReadinessStatus time.Duration) (*resourcemetrics.Metric, error)
}

ResourceGatherer allows retrieval of resource metrics.