magnum

README

Cluster Autoscaler for OpenStack Magnum

The cluster autoscaler for Magnum scales worker nodes within any specified nodegroup. It will run as a Deployment in your cluster. This README will go over some of the necessary steps required to get the cluster autoscaler up and running.

Compatibility

• For Magnum Rocky or earlier: cluster autoscaler v1.18 or lower.
• For Magnum Train or later: cluster autoscaler v1.19 or higher.

Cluster autoscaler versions v1.18 and lower will continue to work on Magnum Train and later versions, but will only support the single default node group. No extra node groups should be added to clusters using the cluster autoscaler v1.18 or lower.

Updates

• CA 1.22
  • Allow scaling node groups to 0 nodes, if supported (requires Magnum Wallaby).
• CA 1.19
  • Update to support Magnum node groups (introduced in Magnum Train).
    • Add node group autodiscovery based on the group's role property.
  • Report upcoming/failed nodes so that CA can back off if the OpenStack project quota is being exceeded.
• CA 1.15
  • Initial release.

Permissions and credentials

The autoscaler needs a ServiceAccount with permissions for Kubernetes and requires credentials for interacting with OpenStack.

An example ServiceAccount is given in examples/cluster-autoscaler-svcaccount.yaml.

The credentials for authenticating with OpenStack are stored in a secret and mounted as a file inside the container. examples/cluster-autoscaler-secret can be modified with the contents of your cloud-config. This file can be obtained from your control plane (previously referred to as master) node, in /etc/kubernetes (may be named kube_openstack_config instead of cloud-config).

Autoscaler deployment

The deployment in examples/cluster-autoscaler-deployment.yaml can be used, but the arguments passed to the autoscaler will need to be changed to match your cluster.

Argument	Usage
--cluster-name	The name of your Kubernetes cluster. If there are multiple clusters sharing the same name then the cluster IDs should be used instead.
--cloud-provider	Can be omitted if the autoscaler is built with BUILD_TAGS=magnum, otherwise use --cloud-provider=magnum.
--nodes	Used to select a specific node group to autoscale and constrain its node count. Of the form min:max:NodeGroupName. Can be used multiple times.
--node-group-auto-discovery	See below.

Deployment with helm

Alternatively, the autoscaler can be deployed with the cluster autoscaler helm chart. A minimal values.yaml file looks like:

cloudProvider: "magnum"

magnumClusterName: "cluster name or ID"

autoscalingGroups:
- name: default-worker
  maxSize: 5
  minSize: 1

cloudConfigPath: "/etc/kubernetes/cloud-config"

For running on the control plane (previously referred to as master) node and other suggested settings, see examples/values-example.yaml. To deploy with node group autodiscovery (for cluster autoscaler v1.19+), see examples/values-autodiscovery.yaml.

Node group auto discovery

Instead of using --nodes to select specific node groups by name, node group auto discovery can be used to to let the autoscaler find which node groups to autoscale by itself, by checking every node group in the cluster against a set of conditions.

The first condition is given in the auto discovery parameter, to select one or more node group roles which should be autoscalable.

--node-group-auto-discovery=magnum:role=worker,autoscaling

The above configuration means that for the Magnum provider, any node group which has a role of "worker" or "autoscaling" should be managed by the cluster autoscaler. The auto discovery parameter can be used multiple times, so the same configuration could be written as:

--node-group-auto-discovery=magnum:role=worker
--node-group-auto-discovery=magnum:role=autoscaling

The second condition is that the node group must have a maximum node count set in Magnum. This can be done using the following command:

$ openstack coe nodegroup update <cluster> <nodegroup> replace /max_node_count=5

which would set the maximum node count to 5 for whichever node group is updated.

By default the min_node_count for a node group is 1, but this can also be changed.

The role of a node group can not be changed after is had been created, but to disable autoscaling for a node group it is enough to unset the maximum node count.

$ openstack coe nodegroup update <cluster> <nodegroup> remove /max_node_count

Notes

The autoscaler will not remove nodes which have non-default kube-system pods. This prevents the node that the autoscaler is running on from being scaled down. If you are deploying the autoscaler into a cluster which already has more than one node, it is best to deploy it onto any node which already has non-default kube-system pods, to minimise the number of nodes which cannot be removed when scaling.

Or, if you are using a Magnum version which supports scheduling on the control plane (previously referred to as master) node, then the example deployment file examples/cluster-autoscaler-deployment-master.yaml can be used.

Documentation

Index

• Constants
• func BuildMagnum(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, ...) cloudprovider.CloudProvider
• type BlockStorageOpts
• type Config
• type LoadBalancerOpts
• type MetadataOpts
• type MyDuration
  • func (d *MyDuration) UnmarshalText(text []byte) error
• type NodeRef
• type RouterOpts

Constants

const (
	// GPULabel is the label added to nodes with GPU resource.
	GPULabel = "magnum.openstack.org/gpu"
)

Functions

func BuildMagnum

func BuildMagnum(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter) cloudprovider.CloudProvider

BuildMagnum is called by the autoscaler to build a magnum cloud provider.

The magnumManager is created here, and the initial node groups are created based on the static or auto discovery specs provided via the command line parameters.

Types

type BlockStorageOpts

type BlockStorageOpts struct {
	BSVersion             string `gcfg:"bs-version"`        // overrides autodetection. v1 or v2. Defaults to auto
	TrustDevicePath       bool   `gcfg:"trust-device-path"` // See Issue #33128
	IgnoreVolumeAZ        bool   `gcfg:"ignore-volume-az"`
	NodeVolumeAttachLimit int    `gcfg:"node-volume-attach-limit"` // override volume attach limit for Cinder. Default is : 256
}

BlockStorageOpts is used to talk to Cinder service

type Config

type Config struct {
	Global struct {
		AuthURL         string `gcfg:"auth-url"`
		Username        string
		UserID          string `gcfg:"user-id"`
		Password        string
		TenantID        string `gcfg:"tenant-id"`
		TenantName      string `gcfg:"tenant-name"`
		TrustID         string `gcfg:"trust-id"`
		DomainID        string `gcfg:"domain-id"`
		DomainName      string `gcfg:"domain-name"`
		Region          string
		CAFile          string `gcfg:"ca-file"`
		TLSInsecure     string `gcfg:"tls-insecure"`
		SecretName      string `gcfg:"secret-name"`
		SecretNamespace string `gcfg:"secret-namespace"`
	}
	LoadBalancer LoadBalancerOpts
	BlockStorage BlockStorageOpts
	Route        RouterOpts
	Metadata     MetadataOpts
}

Config is used to read and store information from the cloud configuration file

Taken from kubernetes/pkg/cloudprovider/providers/openstack/openstack.go LoadBalancer, BlockStorage, Route, Metadata are not needed for the autoscaler, but are kept so that if a cloud-config file with those sections is provided then the parsing will not fail.

type LoadBalancerOpts

type LoadBalancerOpts struct {
	LBVersion            string     `gcfg:"lb-version"`          // overrides autodetection. Only support v2.
	UseOctavia           bool       `gcfg:"use-octavia"`         // uses Octavia V2 service catalog endpoint
	SubnetID             string     `gcfg:"subnet-id"`           // overrides autodetection.
	FloatingNetworkID    string     `gcfg:"floating-network-id"` // If specified, will create floating ip for loadbalancer, or do not create floating ip.
	LBMethod             string     `gcfg:"lb-method"`           // default to ROUND_ROBIN.
	LBProvider           string     `gcfg:"lb-provider"`
	CreateMonitor        bool       `gcfg:"create-monitor"`
	MonitorDelay         MyDuration `gcfg:"monitor-delay"`
	MonitorTimeout       MyDuration `gcfg:"monitor-timeout"`
	MonitorMaxRetries    uint       `gcfg:"monitor-max-retries"`
	ManageSecurityGroups bool       `gcfg:"manage-security-groups"`
	NodeSecurityGroupIDs []string   // Do not specify, get it automatically when enable manage-security-groups. TODO(FengyunPan): move it into cache
}

LoadBalancerOpts have the options to talk to Neutron LBaaSV2 or Octavia

type MetadataOpts

type MetadataOpts struct {
	SearchOrder    string     `gcfg:"search-order"`
	RequestTimeout MyDuration `gcfg:"request-timeout"`
}

MetadataOpts is used for configuring how to talk to metadata service or config drive

type MyDuration

type MyDuration struct {
	time.Duration
}

MyDuration is the encoding.TextUnmarshaler interface for time.Duration

func (*MyDuration) UnmarshalText

func (d *MyDuration) UnmarshalText(text []byte) error

UnmarshalText is used to convert from text to Duration

type NodeRef

type NodeRef struct {
	Name       string
	SystemUUID string
	ProviderID string
	IsFake     bool
}

NodeRef stores the name, systemUUID and providerID of a node. For refs which are created from fake nodes, IsFake should be true.

type RouterOpts

type RouterOpts struct {
	RouterID string `gcfg:"router-id"` // required
}

RouterOpts is used for Neutron routes