azure

package
v0.0.0-...-12dba09 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 11, 2024 License: Apache-2.0 Imports: 57 Imported by: 20

README

Cluster Autoscaler on Azure

The cluster autoscaler on Azure dynamically scales Kubernetes worker nodes. It runs as a deployment in your cluster.

This README will help you get cluster autoscaler running on your Azure Kubernetes cluster.

Kubernetes Version

Kubernetes v1.10.x or later is required to use cluster autoscaler on Azure. See the "Releases" section in the README for more information.

CA Version

Cluster autoscaler v1.2.x or later is required for Azure. See the "Releases" section in the README for more information.

NOTE: In the deployment manifests referenced below, be sure to replace the {{ ca_version }} placeholder with an actual release, such as v1.14.2.

Permissions

Get Azure credentials by running the following Azure CLI command:

# replace <subscription-id> with yours.
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/<subscription-id>" --output json

This will create a new service principal with "Contributor" role scoped to your subscription. Save the JSON output, because it will be needed to configure the cluster autoscaler deployment in the next step.

Scaling a VMSS node group to and from 0

If you are using nodeSelector, you need to tag the VMSS with a node-template key "k8s.io_cluster-autoscaler_node-template_label_" for using labels and "k8s.io_cluster-autoscaler_node-template_taint_" if you are using taints.

Note that these tags use the pipe _ character compared to a forward slash due to Azure tag name restrictions.

Examples
Labels

To add the label of foo=bar to a node from a VMSS pool, you would add the following tag to the VMSS k8s.io_cluster-autoscaler_node-template_label_foo: bar.

You can also use forward slashes in the labels by setting them as an underscore in the tag name. For example to add the label of k8s.io/foo=bar to a node from a VMSS pool, you would add the following tag to the VMSS k8s.io_cluster-autoscaler_node-template_label_k8s.io_foo: bar. To encode a tag name containing an underscore, use "~2" (eg. "cpu~2arch" gives "cpu_arch").

Taints

To add the taint of foo=bar:NoSchedule to a node from a VMSS pool, you would add the following tag to the VMSS k8s.io_cluster-autoscaler_node-template_taint_foo: bar:NoSchedule.

You can also use forward slashes in taints by setting them as an underscore in the tag name. For example to add the taint of k8s.io/foo=bar:NoSchedule to a node from a VMSS pool, you would add the following tag to the VMSS k8s.io_cluster-autoscaler_node-template_taint_k8s.io_foo: bar:NoSchedule. To encode a taint name containing an underscore, use "~2".

Resources

When scaling from an empty VM Scale Set (0 instances), Cluster Autoscaler will evaluate the provided resources (cpu, memory, ephemeral-storage) based on that VM Scale Set's backing instance type. This can be overridden (for instance, to account for system reserved resources) by specifying capacities with VMSS tags, formated as: k8s.io_cluster-autoscaler_node-template_resources_<resource name>: <resource value>. For instance:

k8s.io_cluster-autoscaler_node-template_resources_cpu: 3800m
k8s.io_cluster-autoscaler_node-template_resources_memory: 11Gi

NOTE: GPU autoscaling consideration on VMSS : In case of scale set of GPU nodes, kubelet node label accelerator have to be added to node provisionned to make GPU scaling works.

Autoscaling options

Some autoscaling options can be defined per VM Scale Set, with tags. Those tags values have the format as the respective cluster-autoscaler flags they override: floats or durations encoded as strings.

Supported options tags (with example values) are:

# overrides --scale-down-utilization-threshold global value for that specific VM Scale Set
k8s.io_cluster-autoscaler_node-template_autoscaling-options_scaledownutilizationthreshold: "0.5"

# overrides --scale-down-gpu-utilization-threshold global value for that specific VM Scale Set
k8s.io_cluster-autoscaler_node-template_autoscaling-options_scaledowngpuutilizationthreshold: "0.5"

# overrides --scale-down-unneeded-time global value for that specific VM Scale Set
k8s.io_cluster-autoscaler_node-template_autoscaling-options_scaledownunneededtime: "10m0s"

# overrides --scale-down-unready-time global value for that specific VM Scale Set
k8s.io_cluster-autoscaler_node-template_autoscaling-options_scaledownunreadytime: "20m0s"

Deployment manifests

Cluster autoscaler supports four Kubernetes cluster options on Azure:

  • vmss: Autoscale VMSS instances by setting the Azure cloud provider's vmType parameter to vmss or to an empty string. This supports clusters deployed with aks-engine.
  • standard: Autoscale VMAS (Virtual Machine Availability Set) VMs by setting the Azure cloud provider's vmType parameter to standard. This supports clusters deployed with aks-engine.

NOTE: only the vmss option supports scaling down to zero nodes.

NOTE: The subscriptionID parameter is optional. When skipped, the subscription will be fetched from the instance metadata.

VMSS deployment

Prerequisites:

  • Get Azure credentials from the Permissions step above.
  • Get the name of the VM scale set associated with the cluster's node pool. You can find this in the Azure Portal or with the az vmss list command.

Make a copy of cluster-autoscaler-vmss.yaml. Fill in the placeholder values for the cluster-autoscaler-azure secret data by base64-encoding each of your Azure credential fields.

  • ClientID: <base64-encoded-client-id>
  • ClientSecret: <base64-encoded-client-secret>
  • ResourceGroup: <base64-encoded-resource-group>
  • SubscriptionID: <base64-encoded-subscription-id>
  • TenantID: <base64-encoded-tenant-id>

NOTE: Use a command such as echo $CLIENT_ID | base64 to encode each of the fields above.

In the cluster-autoscaler spec, find the image: field and replace {{ ca_version }} with a specific cluster autoscaler release.

Auto-Discovery Setup

To run a cluster-autoscaler which auto-discovers VMSSs with nodes use the --node-group-auto-discovery flag. For example, --node-group-auto-discovery=label:cluster-autoscaler-enabled=true,cluster-autoscaler-name=<YOUR CLUSTER NAME> will find the VMSSs tagged with those tags containing those values.

Note that:

  • It is recommended to use a second tag like cluster-autoscaler-name=<YOUR CLUSTER NAME> when cluster-autoscaler-enabled=true is used across many clusters to prevent VMSSs from different clusters recognized as the node groups
  • There are no --nodes flags passed to cluster-autoscaler because the node groups are automatically discovered by tags
  • No min/max values are provided when using Auto-Discovery, cluster-autoscaler will detect the "min" and "max" tags on the VMSS resource in Azure, adjusting the desired number of nodes within these limits.
kubectl apply -f examples/cluster-autoscaler-autodiscover.yaml
Explicit setup

Below that, in the command: section, update the --nodes= arguments to reference your node limits and VMSS name. For example, if node pool "k8s-nodepool-1-vmss" should scale from 1 to 10 nodes:

        - --nodes=1:10:k8s-nodepool-1-vmss

or to autoscale multiple VM scale sets:

        - --nodes=1:10:k8s-nodepool-1-vmss
        - --nodes=1:10:k8s-nodepool-2-vmss

Note that it doesn't mean the number of nodes in nodepool is restricted in the range from 1 to 10. It means when ca is downscaling (upscaling) the nodepool, it will never break the limit of 1 (10). If the current node pool size is lower than the specified minimum or greater than the specified maximum when you enable autoscaling, the autoscaler waits to take effect until a new node is needed in the node pool or until a node can be safely deleted from the node pool.

To allow scaling similar node pools simultaneously, or when using separate node groups per zone and to keep nodes balanced across zones, use the --balance-similar-node-groups flag (default false). Add it to the command section to enable it:

        - --balance-similar-node-groups=true

See the FAQ for more details.

Save the updated deployment manifest, then deploy cluster-autoscaler by running:

kubectl create -f cluster-autoscaler-vmss.yaml

To run a cluster autoscaler pod on a control plane (previously referred to as master) node, the deployment should tolerate the master taint, and nodeSelector should be used to schedule pods. Use cluster-autoscaler-vmss-control-plane.yaml in this case.

To run a cluster autoscaler pod with Azure managed service identity (MSI), use cluster-autoscaler-vmss-msi.yaml instead.

Azure API Throttling

Azure has hard limits on the number of read and write requests against Azure APIs per subscription, per region. Running lots of clusters in a single subscription, or running a single large, dynamic cluster in a subscription can produce side effects that exceed the number of calls permitted within a given time window for a particular category of requests. See the following documents for more detail on Azure API throttling in general:

Given the dynamic nature of cluster autoscaler, it can be a trigger for hitting those rate limits on the subscriptions. This in turn can affect other components running in the cluster that depend on Azure APIs such as kube-controller-manager.

When using K8s versions older than v1.18, we recommend using at least v.1.17.5, v1.16.9, v1.15.12 which include various improvements on the cloud-provider side that have an impact on the number of API calls during scale down operations.

As for CA versions older than 1.18, we recommend using at least v.1.17.2, v1.16.5, v1.15.6.

In addition, cluster-autoscaler exposes a AZURE_VMSS_CACHE_TTL environment variable which controls the rate of GetVMScaleSet being made. By default, this is 15 seconds but setting this to a higher value such as 60 seconds can protect against API throttling. The caches used are proactively incremented and decremented with the scale up and down operations and this higher value doesn't have any noticeable impact on performance. Note that the value is in seconds

Config Name Default Environment Variable Cloud Config File
VmssCacheTTL 60 AZURE_VMSS_CACHE_TTL vmssCacheTTL

The AZURE_VMSS_VMS_CACHE_TTL environment variable affects the GetScaleSetVms (VMSS VM List) calls rate. The default value is 300 seconds. A configurable jitter (AZURE_VMSS_VMS_CACHE_JITTER environment variable, default 0) expresses the maximum number of second that will be subtracted from that initial VMSS cache TTL after a new VMSS is discovered by the cluster-autoscaler: this can prevent a dogpile effect on clusters having many VMSS.

Config Name Default Environment Variable Cloud Config File
vmssVmsCacheTTL 300 AZURE_VMSS_VMS_CACHE_TTL vmssVmsCacheTTL
vmssVmsCacheJitter 0 AZURE_VMSS_VMS_CACHE_JITTER vmssVmsCacheJitter

The AZURE_ENABLE_DYNAMIC_INSTANCE_LIST environment variable enables workflow that fetched SKU information dynamically using SKU API calls. By default, it uses static list of SKUs.

Config Name Default Environment Variable Cloud Config File
enableDynamicInstanceList false AZURE_ENABLE_DYNAMIC_INSTANCE_LIST enableDynamicInstanceList

The AZURE_ENABLE_VMSS_FLEX environment variable enables VMSS Flex support. By default, support is disabled.

Config Name Default Environment Variable Cloud Config File
enableVmssFlex false AZURE_ENABLE_VMSS_FLEX enableVmssFlex

When using K8s 1.18 or higher, it is also recommended to configure backoff and retries on the client as described here

Standard deployment

Prerequisites:

  • Get Azure credentials from the Permissions step above.
  • Get the name of the initial Azure deployment resource for the cluster. You can find this in the Azure Portal or with the az deployment list command. If there are multiple deployments, get the name of the first one.

Make a copy of cluster-autoscaler-standard-control-plane.yaml. Fill in the placeholder values for the cluster-autoscaler-azure secret data by base64-encoding each of your Azure credential fields.

  • ClientID: <base64-encoded-client-id>
  • ClientSecret: <base64-encoded-client-secret>
  • ResourceGroup: <base64-encoded-resource-group>
  • SubscriptionID: <base64-encoded-subscription-id>
  • TenantID: <base64-encoded-tenant-id>
  • Deployment: <base64-encoded-azure-initial-deployment-name>

NOTE: Use a command such as echo $CLIENT_ID | base64 to encode each of the fields above.

In the cluster-autoscaler spec, find the image: field and replace {{ ca_version }} with a specific cluster autoscaler release.

Below that, in the command: section, update the --nodes= arguments to reference your node limits and node pool name (tips: node pool name is NOT availability set name, e.g., the corresponding node pool name of the availability set agentpool1-availabilitySet-xxxxxxxx would be agentpool1). For example, if node pool "k8s-nodepool-1" should scale from 1 to 10 nodes:

        - --nodes=1:10:k8s-nodepool-1

or to autoscale multiple VM scale sets:

        - --nodes=1:10:k8s-nodepool-1
        - --nodes=1:10:k8s-nodepool-2

Create the Azure deploy parameters secret cluster-autoscaler-azure-deploy-parameters by running:

kubectl -n kube-system create secret generic cluster-autoscaler-azure-deploy-parameters --from-file=deploy-parameters=./_output/<your-output-path>/azuredeploy.parameters.json

Then deploy cluster-autoscaler by running:

kubectl create -f cluster-autoscaler-standard-control-plane.yaml

To run a cluster autoscaler pod with Azure managed service identity (MSI), use cluster-autoscaler-standard-msi.yaml instead.

WARNING: Cluster autoscaler depends on user-provided deployment parameters to provision new nodes. After upgrading your Kubernetes cluster, cluster autoscaler must also be redeployed with new parameters to prevent provisioning nodes with an old version.

AKS Autoscaler

Node Pool Autoscaling is a first class feature of your AKS cluster. The option to enable cluster autoscaler is available in the Azure Portal or with the Azure CLI:

az aks create \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --kubernetes-version 1.25.11 \
  --node-count 1 \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 3

Please see the AKS autoscaler documentation for details.

Rate limit and back-off retries

The new version of Azure client supports rate limit and back-off retries when the cluster hits the throttling issue. These can be set by either environment variables, or cloud config file. With config file, defaults values are false or 0.

Config Name Default Environment Variable Cloud Config File
CloudProviderBackoff false ENABLE_BACKOFF cloudProviderBackoff
CloudProviderBackoffRetries 6 BACKOFF_RETRIES cloudProviderBackoffRetries
CloudProviderBackoffExponent 1.5 BACKOFF_EXPONENT cloudProviderBackoffExponent
CloudProviderBackoffDuration 5 BACKOFF_DURATION cloudProviderBackoffDuration
CloudProviderBackoffJitter 1.0 BACKOFF_JITTER cloudProviderBackoffJitter
CloudProviderRateLimit * false CLOUD_PROVIDER_RATE_LIMIT cloudProviderRateLimit
CloudProviderRateLimitQPS * 1 RATE_LIMIT_READ_QPS cloudProviderRateLimitQPS
CloudProviderRateLimitBucket * 5 RATE_LIMIT_READ_BUCKETS cloudProviderRateLimitBucket
CloudProviderRateLimitQPSWrite * 1 RATE_LIMIT_WRITE_QPS cloudProviderRateLimitQPSWrite
CloudProviderRateLimitBucketWrite * 5 RATE_LIMIT_WRITE_BUCKETS cloudProviderRateLimitBucketWrite

NOTE: * These rate limit configs can be set per-client. Customizing QPS and Bucket through environment variables per client is not supported.

Documentation

Index

Constants

View Source 
const (
	// GPULabel is the label added to nodes with GPU resource.
	GPULabel = "accelerator"
)

Variables

View Source 
var GetVMSSTypeDynamically = func(template compute.VirtualMachineScaleSet, azCache *azureCache) (InstanceType, error) {
	ctx := context.Background()
	var vmssType InstanceType

	sku, err := azCache.GetSKU(ctx, *template.Sku.Name, *template.Location)
	if err != nil {

		promoRe := regexp.MustCompile(`(?i)_promo`)
		skuName := promoRe.ReplaceAllString(*template.Sku.Name, "")
		if skuName != *template.Sku.Name {
			klog.V(1).Infof("No exact match found for %q, checking standard type %q. Error %v", *template.Sku.Name, skuName, err)
			sku, err = azCache.GetSKU(ctx, skuName, *template.Location)
		}
		if err != nil {
			return vmssType, fmt.Errorf("instance type %q not supported. Error %v", *template.Sku.Name, err)
		}
	}

	vmssType.VCPU, err = sku.VCPU()
	if err != nil {
		klog.V(1).Infof("Failed to parse vcpu from sku %q %v", *template.Sku.Name, err)
		return vmssType, err
	}
	gpu, err := getGpuFromSku(sku)
	if err != nil {
		klog.V(1).Infof("Failed to parse gpu from sku %q %v", *template.Sku.Name, err)
		return vmssType, err
	}
	vmssType.GPU = gpu

	memoryGb, err := sku.Memory()
	if err != nil {
		klog.V(1).Infof("Failed to parse memoryMb from sku %q %v", *template.Sku.Name, err)
		return vmssType, err
	}
	vmssType.MemoryMb = int64(memoryGb) * 1024

	return vmssType, nil
}

GetVMSSTypeDynamically fetched vmss instance information using sku api calls. It is declared as a variable for testing purpose.

View Source 
var GetVMSSTypeStatically = func(template compute.VirtualMachineScaleSet) (*InstanceType, error) {
	var vmssType *InstanceType

	for k := range InstanceTypes {
		if strings.EqualFold(k, *template.Sku.Name) {
			vmssType = InstanceTypes[k]
			break
		}
	}

	promoRe := regexp.MustCompile(`(?i)_promo`)
	if promoRe.MatchString(*template.Sku.Name) {
		if vmssType == nil {

			klog.V(4).Infof("No exact match found for %s, checking standard types", *template.Sku.Name)
			skuName := promoRe.ReplaceAllString(*template.Sku.Name, "")
			for k := range InstanceTypes {
				if strings.EqualFold(k, skuName) {
					vmssType = InstanceTypes[k]
					break
				}
			}
		}
	}
	if vmssType == nil {
		return vmssType, fmt.Errorf("instance type %q not supported", *template.Sku.Name)
	}
	return vmssType, nil
}

GetVMSSTypeStatically uses static list of vmss generated at azure_instance_types.go to fetch vmss instance information. It is declared as a variable for testing purpose.

View Source 
var InstanceTypes = map[string]*InstanceType{}/* 741 elements not displayed */

InstanceTypes is a map of azure resources

View Source 
var (
	// NvidiaEnabledSKUs represents a list of NVIDIA gpus.
	// If a new GPU sku becomes available, add a key to this map, but only if you have a confirmation
	//   that we have an agreement with NVIDIA for this specific gpu.
	NvidiaEnabledSKUs = map[string]bool{

		"standard_nc6":   true,
		"standard_nc12":  true,
		"standard_nc24":  true,
		"standard_nc24r": true,

		"standard_nv6":      true,
		"standard_nv12":     true,
		"standard_nv12s_v3": true,
		"standard_nv24":     true,
		"standard_nv24s_v3": true,
		"standard_nv24r":    true,
		"standard_nv48s_v3": true,

		"standard_nd6s":   true,
		"standard_nd12s":  true,
		"standard_nd24s":  true,
		"standard_nd24rs": true,

		"standard_nc6s_v2":   true,
		"standard_nc12s_v2":  true,
		"standard_nc24s_v2":  true,
		"standard_nc24rs_v2": true,

		"standard_nc6s_v3":   true,
		"standard_nc12s_v3":  true,
		"standard_nc24s_v3":  true,
		"standard_nc24rs_v3": true,
		"standard_nd40s_v3":  true,
		"standard_nd40rs_v2": true,

		"standard_nc4as_t4_v3":  true,
		"standard_nc8as_t4_v3":  true,
		"standard_nc16as_t4_v3": true,
		"standard_nc64as_t4_v3": true,

		"standard_nd96asr_v4":       true,
		"standard_nd112asr_a100_v4": true,
		"standard_nd120asr_a100_v4": true,

		"standard_nd96amsr_a100_v4":  true,
		"standard_nd112amsr_a100_v4": true,
		"standard_nd120amsr_a100_v4": true,

		"standard_nc24ads_a100_v4": true,
		"standard_nc48ads_a100_v4": true,
		"standard_nc96ads_a100_v4": true,
	}
)

Functions

func BuildAzure 

func BuildAzure(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter) cloudprovider.CloudProvider

BuildAzure builds Azure cloud provider, manager etc.

func BuildAzureCloudProvider 

func BuildAzureCloudProvider(azureManager *AzureManager, resourceLimiter *cloudprovider.ResourceLimiter) (cloudprovider.CloudProvider, error)

BuildAzureCloudProvider creates new AzureCloudProvider

func GetVMNameIndex 

func GetVMNameIndex(osType compute.OperatingSystemTypes, vmName string) (int, error)

GetVMNameIndex return the index of VM in the node pools.

func ParseLabelAutoDiscoverySpecs 

func ParseLabelAutoDiscoverySpecs(o cloudprovider.NodeGroupDiscoveryOptions) ([]labelAutoDiscoveryConfig, error)

ParseLabelAutoDiscoverySpecs returns any provided NodeGroupAutoDiscoverySpecs parsed into configuration appropriate for node group autodiscovery.

Types

type AgentPool 

type AgentPool struct {
	// contains filtered or unexported fields
}

AgentPool implements NodeGroup interface for agent pools deployed by aks-engine.

func NewAgentPool 

func NewAgentPool(spec *dynamic.NodeGroupSpec, az *AzureManager) (*AgentPool, error)

NewAgentPool creates a new AgentPool.

func (*AgentPool) AtomicIncreaseSize 

func (as *AgentPool) AtomicIncreaseSize(delta int) error

AtomicIncreaseSize is not implemented.

func (*AgentPool) Autoprovisioned 

func (as *AgentPool) Autoprovisioned() bool

Autoprovisioned returns true if the node group is autoprovisioned.

func (*AgentPool) Belongs 

func (as *AgentPool) Belongs(node *apiv1.Node) (bool, error)

Belongs returns true if the given node belongs to the NodeGroup.

func (*AgentPool) Create 

func (as *AgentPool) Create() (cloudprovider.NodeGroup, error)

Create creates the node group on the cloud provider side.

func (*AgentPool) Debug 

func (as *AgentPool) Debug() string

Debug returns a debug string for the agent pool.

func (*AgentPool) DecreaseTargetSize 

func (as *AgentPool) DecreaseTargetSize(delta int) error

DecreaseTargetSize decreases the target size of the node group. This function doesn't permit to delete any existing node and can be used only to reduce the request for new nodes that have not been yet fulfilled. Delta should be negative. It is assumed that cloud provider will not delete the existing nodes if the size when there is an option to just decrease the target.

func (*AgentPool) Delete 

func (as *AgentPool) Delete() error

Delete deletes the node group on the cloud provider side. This will be executed only for autoprovisioned node groups, once their size drops to 0.

func (*AgentPool) DeleteInstances 

func (as *AgentPool) DeleteInstances(instances []*azureRef) error

DeleteInstances deletes the given instances. All instances must be controlled by the same ASG.

func (*AgentPool) DeleteNodes 

func (as *AgentPool) DeleteNodes(nodes []*apiv1.Node) error

DeleteNodes deletes the nodes from the group.

func (*AgentPool) Exist 

func (as *AgentPool) Exist() bool

Exist checks if the node group really exists on the cloud provider side. Allows to tell the theoretical node group from the real one.

func (*AgentPool) GetKey 

func (m *AgentPool) GetKey() string

GetKey returns key of the given azure reference.

func (*AgentPool) GetOptions 

func (as *AgentPool) GetOptions(defaults config.NodeGroupAutoscalingOptions) (*config.NodeGroupAutoscalingOptions, error)

GetOptions returns NodeGroupAutoscalingOptions that should be used for this particular NodeGroup. Returning a nil will result in using default options.

func (*AgentPool) GetVMIndexes 

func (as *AgentPool) GetVMIndexes() ([]int, map[int]string, error)

GetVMIndexes gets indexes of all virtual machines belonging to the agent pool.

func (*AgentPool) Id 

func (as *AgentPool) Id() string

Id returns AgentPool id.

func (*AgentPool) IncreaseSize 

func (as *AgentPool) IncreaseSize(delta int) error

IncreaseSize increases agent pool size

func (*AgentPool) MaxSize 

func (as *AgentPool) MaxSize() int

MaxSize returns maximum size of the node group.

func (*AgentPool) MinSize 

func (as *AgentPool) MinSize() int

MinSize returns minimum size of the node group.

func (*AgentPool) Nodes 

func (as *AgentPool) Nodes() ([]cloudprovider.Instance, error)

Nodes returns a list of all nodes that belong to this node group.

func (*AgentPool) String 

func (m *AgentPool) String() string

String is represented by calling GetKey()

func (*AgentPool) TargetSize 

func (as *AgentPool) TargetSize() (int, error)

TargetSize returns the current TARGET size of the node group. It is possible that the number is different from the number of nodes registered in Kubernetes.

func (*AgentPool) TemplateNodeInfo 

func (as *AgentPool) TemplateNodeInfo() (*schedulerframework.NodeInfo, error)

TemplateNodeInfo returns a node template for this agent pool.

type AzUtil 

type AzUtil struct {
	// contains filtered or unexported fields
}

AzUtil consists of utility functions which utilizes clients to different services. Since they span across various clients they cannot be fitted into individual client structs so adding them here.

func (*AzUtil) DeleteBlob 

func (util *AzUtil) DeleteBlob(accountName, vhdContainer, vhdBlob string) error

DeleteBlob deletes the blob using the storage client.

func (*AzUtil) DeleteVirtualMachine 

func (util *AzUtil) DeleteVirtualMachine(rg string, name string) error

DeleteVirtualMachine deletes a VM and any associated OS disk

type AzureCloudProvider 

type AzureCloudProvider struct {
	// contains filtered or unexported fields
}

AzureCloudProvider provides implementation of CloudProvider interface for Azure.

func (*AzureCloudProvider) Cleanup 

func (azure *AzureCloudProvider) Cleanup() error

Cleanup stops the go routine that is handling the current view of the ASGs in the form of a cache

func (*AzureCloudProvider) GPULabel 

func (azure *AzureCloudProvider) GPULabel() string

GPULabel returns the label added to nodes with GPU resource.

func (*AzureCloudProvider) GetAvailableGPUTypes 

func (azure *AzureCloudProvider) GetAvailableGPUTypes() map[string]struct{}

GetAvailableGPUTypes return all available GPU types cloud provider supports

func (*AzureCloudProvider) GetAvailableMachineTypes 

func (azure *AzureCloudProvider) GetAvailableMachineTypes() ([]string, error)

GetAvailableMachineTypes get all machine types that can be requested from the cloud provider.

func (*AzureCloudProvider) GetNodeGpuConfig 

func (azure *AzureCloudProvider) GetNodeGpuConfig(node *apiv1.Node) *cloudprovider.GpuConfig

GetNodeGpuConfig returns the label, type and resource name for the GPU added to node. If node doesn't have any GPUs, it returns nil.

func (*AzureCloudProvider) GetResourceLimiter 

func (azure *AzureCloudProvider) GetResourceLimiter() (*cloudprovider.ResourceLimiter, error)

GetResourceLimiter returns struct containing limits (max, min) for resources (cores, memory etc.).

func (*AzureCloudProvider) HasInstance 

func (azure *AzureCloudProvider) HasInstance(*apiv1.Node) (bool, error)

HasInstance returns whether a given node has a corresponding instance in this cloud provider

func (*AzureCloudProvider) Name 

func (azure *AzureCloudProvider) Name() string

Name returns name of the cloud provider.

func (*AzureCloudProvider) NewNodeGroup 

func (azure *AzureCloudProvider) NewNodeGroup(machineType string, labels map[string]string, systemLabels map[string]string,
	taints []apiv1.Taint, extraResources map[string]resource.Quantity) (cloudprovider.NodeGroup, error)

NewNodeGroup builds a theoretical node group based on the node definition provided. The node group is not automatically created on the cloud provider side. The node group is not returned by NodeGroups() until it is created.

func (*AzureCloudProvider) NodeGroupForNode 

func (azure *AzureCloudProvider) NodeGroupForNode(node *apiv1.Node) (cloudprovider.NodeGroup, error)

NodeGroupForNode returns the node group for the given node.

func (*AzureCloudProvider) NodeGroups 

func (azure *AzureCloudProvider) NodeGroups() []cloudprovider.NodeGroup

NodeGroups returns all node groups configured for this cloud provider.

func (*AzureCloudProvider) Pricing 

func (azure *AzureCloudProvider) Pricing() (cloudprovider.PricingModel, errors.AutoscalerError)

Pricing returns pricing model for this cloud provider or error if not available.

func (*AzureCloudProvider) Refresh 

func (azure *AzureCloudProvider) Refresh() error

Refresh is called before every main loop and can be used to dynamically update cloud provider state. In particular the list of node groups returned by NodeGroups can change as a result of CloudProvider.Refresh().

type AzureManager 

type AzureManager struct {
	// contains filtered or unexported fields
}

AzureManager handles Azure communication and data caching.

func CreateAzureManager 

func CreateAzureManager(configReader io.Reader, discoveryOpts cloudprovider.NodeGroupDiscoveryOptions) (*AzureManager, error)

CreateAzureManager creates Azure Manager object to work with Azure.

func (*AzureManager) Cleanup 

func (m *AzureManager) Cleanup()

Cleanup the cache.

func (*AzureManager) GetNodeGroupForInstance 

func (m *AzureManager) GetNodeGroupForInstance(instance *azureRef) (cloudprovider.NodeGroup, error)

GetNodeGroupForInstance returns the NodeGroup of the given Instance

func (*AzureManager) GetScaleSetOptions 

func (m *AzureManager) GetScaleSetOptions(scaleSetName string, defaults config.NodeGroupAutoscalingOptions) *config.NodeGroupAutoscalingOptions

GetScaleSetOptions parse options extracted from VMSS tags and merges them with provided defaults

func (*AzureManager) Refresh 

func (m *AzureManager) Refresh() error

Refresh is called before every main loop and can be used to dynamically update cloud provider state. In particular the list of node groups returned by NodeGroups can change as a result of CloudProvider.Refresh().

func (*AzureManager) RegisterNodeGroup 

func (m *AzureManager) RegisterNodeGroup(nodeGroup cloudprovider.NodeGroup) bool

RegisterNodeGroup registers an a NodeGroup.

func (*AzureManager) UnregisterNodeGroup 

func (m *AzureManager) UnregisterNodeGroup(nodeGroup cloudprovider.NodeGroup) bool

UnregisterNodeGroup unregisters a NodeGroup.

type CloudProviderRateLimitConfig 

type CloudProviderRateLimitConfig struct {
	// The default rate limit config options.
	azclients.RateLimitConfig

	// Rate limit config for each clients. Values would override default settings above.
	InterfaceRateLimit              *azclients.RateLimitConfig `json:"interfaceRateLimit,omitempty" yaml:"interfaceRateLimit,omitempty"`
	VirtualMachineRateLimit         *azclients.RateLimitConfig `json:"virtualMachineRateLimit,omitempty" yaml:"virtualMachineRateLimit,omitempty"`
	StorageAccountRateLimit         *azclients.RateLimitConfig `json:"storageAccountRateLimit,omitempty" yaml:"storageAccountRateLimit,omitempty"`
	DiskRateLimit                   *azclients.RateLimitConfig `json:"diskRateLimit,omitempty" yaml:"diskRateLimit,omitempty"`
	VirtualMachineScaleSetRateLimit *azclients.RateLimitConfig `json:"virtualMachineScaleSetRateLimit,omitempty" yaml:"virtualMachineScaleSetRateLimit,omitempty"`
	KubernetesServiceRateLimit      *azclients.RateLimitConfig `json:"kubernetesServiceRateLimit,omitempty" yaml:"kubernetesServiceRateLimit,omitempty"`
}

CloudProviderRateLimitConfig indicates the rate limit config for each clients.

type Config 

type Config struct {
	CloudProviderRateLimitConfig

	Cloud          string `json:"cloud" yaml:"cloud"`
	Location       string `json:"location" yaml:"location"`
	TenantID       string `json:"tenantId" yaml:"tenantId"`
	SubscriptionID string `json:"subscriptionId" yaml:"subscriptionId"`
	ResourceGroup  string `json:"resourceGroup" yaml:"resourceGroup"`
	VMType         string `json:"vmType" yaml:"vmType"`

	// AuthMethod determines how to authorize requests for the Azure
	// cloud. Valid options are "principal" (= the traditional
	// service principle approach) and "cli" (= load az command line
	// config file). The default is "principal".
	AuthMethod string `json:"authMethod" yaml:"authMethod"`

	AADClientID                  string `json:"aadClientId" yaml:"aadClientId"`
	AADClientSecret              string `json:"aadClientSecret" yaml:"aadClientSecret"`
	AADClientCertPath            string `json:"aadClientCertPath" yaml:"aadClientCertPath"`
	AADClientCertPassword        string `json:"aadClientCertPassword" yaml:"aadClientCertPassword"`
	AADFederatedTokenFile        string `json:"aadFederatedTokenFile" yaml:"aadFederatedTokenFile"`
	UseManagedIdentityExtension  bool   `json:"useManagedIdentityExtension" yaml:"useManagedIdentityExtension"`
	UseWorkloadIdentityExtension bool   `json:"useWorkloadIdentityExtension" yaml:"useWorkloadIdentityExtension"`
	UserAssignedIdentityID       string `json:"userAssignedIdentityID" yaml:"userAssignedIdentityID"`

	// Configs only for standard vmType (agent pools).
	Deployment           string                 `json:"deployment" yaml:"deployment"`
	DeploymentParameters map[string]interface{} `json:"deploymentParameters" yaml:"deploymentParameters"`

	// VMSS metadata cache TTL in seconds, only applies for vmss type
	VmssCacheTTL int64 `json:"vmssCacheTTL" yaml:"vmssCacheTTL"`

	// VMSS instances cache TTL in seconds, only applies for vmss type
	VmssVmsCacheTTL int64 `json:"vmssVmsCacheTTL" yaml:"vmssVmsCacheTTL"`

	// Jitter in seconds subtracted from the VMSS cache TTL before the first refresh
	VmssVmsCacheJitter int `json:"vmssVmsCacheJitter" yaml:"vmssVmsCacheJitter"`

	// number of latest deployments that will not be deleted
	MaxDeploymentsCount int64 `json:"maxDeploymentsCount" yaml:"maxDeploymentsCount"`

	// Enable exponential backoff to manage resource request retries
	CloudProviderBackoff         bool    `json:"cloudProviderBackoff,omitempty" yaml:"cloudProviderBackoff,omitempty"`
	CloudProviderBackoffRetries  int     `json:"cloudProviderBackoffRetries,omitempty" yaml:"cloudProviderBackoffRetries,omitempty"`
	CloudProviderBackoffExponent float64 `json:"cloudProviderBackoffExponent,omitempty" yaml:"cloudProviderBackoffExponent,omitempty"`
	CloudProviderBackoffDuration int     `json:"cloudProviderBackoffDuration,omitempty" yaml:"cloudProviderBackoffDuration,omitempty"`
	CloudProviderBackoffJitter   float64 `json:"cloudProviderBackoffJitter,omitempty" yaml:"cloudProviderBackoffJitter,omitempty"`

	// EnableForceDelete defines whether to enable force deletion on the APIs
	EnableForceDelete bool `json:"enableForceDelete,omitempty" yaml:"enableForceDelete,omitempty"`

	// EnableDynamicInstanceList defines whether to enable dynamic instance workflow for instance information check
	EnableDynamicInstanceList bool `json:"enableDynamicInstanceList,omitempty" yaml:"enableDynamicInstanceList,omitempty"`

	// EnableVmssFlex defines whether to enable Vmss Flex support or not
	EnableVmssFlex bool `json:"enableVmssFlex,omitempty" yaml:"enableVmssFlex,omitempty"`
}

Config holds the configuration parsed from the --cloud-config flag

func BuildAzureConfig 

func BuildAzureConfig(configReader io.Reader) (*Config, error)

BuildAzureConfig returns a Config object for the Azure clients

func (*Config) TrimSpace 

func (cfg *Config) TrimSpace()

TrimSpace removes all leading and trailing white spaces.

type DeploymentsClient 

type DeploymentsClient interface {
	Get(ctx context.Context, resourceGroupName string, deploymentName string) (result resources.DeploymentExtended, err error)
	List(ctx context.Context, resourceGroupName string, filter string, top *int32) (result []resources.DeploymentExtended, err error)
	ExportTemplate(ctx context.Context, resourceGroupName string, deploymentName string) (result resources.DeploymentExportResult, err error)
	CreateOrUpdate(ctx context.Context, resourceGroupName string, deploymentName string, parameters resources.Deployment) (resp *http.Response, err error)
	Delete(ctx context.Context, resourceGroupName string, deploymentName string) (resp *http.Response, err error)
}

DeploymentsClient defines needed functions for azure network.DeploymentsClient.

type DeploymentsClientMock 

type DeploymentsClientMock struct {
	mock.Mock

	FakeStore map[string]resources.DeploymentExtended
	// contains filtered or unexported fields
}

DeploymentsClientMock mocks for DeploymentsClient.

func (*DeploymentsClientMock) CreateOrUpdate 

func (m *DeploymentsClientMock) CreateOrUpdate(ctx context.Context, resourceGroupName string, deploymentName string, parameters resources.Deployment) (resp *http.Response, err error)

CreateOrUpdate creates or updates the Deployment.

func (*DeploymentsClientMock) Delete 

func (m *DeploymentsClientMock) Delete(ctx context.Context, resourceGroupName, deploymentName string) (resp *http.Response, err error)

Delete deletes the given deployment

func (*DeploymentsClientMock) ExportTemplate 

func (m *DeploymentsClientMock) ExportTemplate(ctx context.Context, resourceGroupName string, deploymentName string) (result resources.DeploymentExportResult, err error)

ExportTemplate exports the deployment's template.

func (*DeploymentsClientMock) Get 

func (m *DeploymentsClientMock) Get(ctx context.Context, resourceGroupName string, deploymentName string) (result resources.DeploymentExtended, err error)

Get gets the DeploymentExtended by deploymentName.

func (*DeploymentsClientMock) List 

func (m *DeploymentsClientMock) List(ctx context.Context, resourceGroupName, filter string, top *int32) (result []resources.DeploymentExtended, err error)

List gets all the deployments for a resource group.

type InstanceType 

type InstanceType struct {
	InstanceType string
	SkuFamily    string
	VCPU         int64
	MemoryMb     int64
	GPU          int64
}

InstanceType is the sepc of Azure instance

type ScaleSet 

type ScaleSet struct {
	// contains filtered or unexported fields
}

ScaleSet implements NodeGroup interface.

func NewScaleSet 

func NewScaleSet(spec *dynamic.NodeGroupSpec, az *AzureManager, curSize int64) (*ScaleSet, error)

NewScaleSet creates a new NewScaleSet.

func (*ScaleSet) AtomicIncreaseSize 

func (scaleSet *ScaleSet) AtomicIncreaseSize(delta int) error

AtomicIncreaseSize is not implemented.

func (*ScaleSet) Autoprovisioned 

func (scaleSet *ScaleSet) Autoprovisioned() bool

Autoprovisioned returns true if the node group is autoprovisioned.

func (*ScaleSet) Belongs 

func (scaleSet *ScaleSet) Belongs(node *apiv1.Node) (bool, error)

Belongs returns true if the given node belongs to the NodeGroup.

func (*ScaleSet) Create 

func (scaleSet *ScaleSet) Create() (cloudprovider.NodeGroup, error)

Create creates the node group on the cloud provider side.

func (*ScaleSet) Debug 

func (scaleSet *ScaleSet) Debug() string

Debug returns a debug string for the Scale Set.

func (*ScaleSet) DecreaseTargetSize 

func (scaleSet *ScaleSet) DecreaseTargetSize(delta int) error

DecreaseTargetSize decreases the target size of the node group. This function doesn't permit to delete any existing node and can be used only to reduce the request for new nodes that have not been yet fulfilled. Delta should be negative. It is assumed that cloud provider will not delete the existing nodes if the size when there is an option to just decrease the target.

func (*ScaleSet) Delete 

func (scaleSet *ScaleSet) Delete() error

Delete deletes the node group on the cloud provider side. This will be executed only for autoprovisioned node groups, once their size drops to 0.

func (*ScaleSet) DeleteInstances 

func (scaleSet *ScaleSet) DeleteInstances(instances []*azureRef, hasUnregisteredNodes bool) error

DeleteInstances deletes the given instances. All instances must be controlled by the same ASG.

func (*ScaleSet) DeleteNodes 

func (scaleSet *ScaleSet) DeleteNodes(nodes []*apiv1.Node) error

DeleteNodes deletes the nodes from the group.

func (*ScaleSet) Exist 

func (scaleSet *ScaleSet) Exist() bool

Exist checks if the node group really exists on the cloud provider side. Allows to tell the theoretical node group from the real one.

func (*ScaleSet) GetFlexibleScaleSetVms 

func (scaleSet *ScaleSet) GetFlexibleScaleSetVms() ([]compute.VirtualMachine, *retry.Error)

GetFlexibleScaleSetVms returns list of nodes for flexible scale set.

func (*ScaleSet) GetKey 

func (m *ScaleSet) GetKey() string

GetKey returns key of the given azure reference.

func (*ScaleSet) GetOptions 

func (scaleSet *ScaleSet) GetOptions(defaults config.NodeGroupAutoscalingOptions) (*config.NodeGroupAutoscalingOptions, error)

GetOptions returns NodeGroupAutoscalingOptions that should be used for this particular NodeGroup. Returning a nil will result in using default options.

func (*ScaleSet) GetScaleSetSize 

func (scaleSet *ScaleSet) GetScaleSetSize() (int64, error)

GetScaleSetSize gets Scale Set size.

func (*ScaleSet) GetScaleSetVms 

func (scaleSet *ScaleSet) GetScaleSetVms() ([]compute.VirtualMachineScaleSetVM, *retry.Error)

GetScaleSetVms returns list of nodes for the given scale set.

func (*ScaleSet) Id 

func (scaleSet *ScaleSet) Id() string

Id returns ScaleSet id.

func (*ScaleSet) IncreaseSize 

func (scaleSet *ScaleSet) IncreaseSize(delta int) error

IncreaseSize increases Scale Set size

func (*ScaleSet) MaxSize 

func (scaleSet *ScaleSet) MaxSize() int

MaxSize returns maximum size of the node group.

func (*ScaleSet) MinSize 

func (scaleSet *ScaleSet) MinSize() int

MinSize returns minimum size of the node group.

func (*ScaleSet) Nodes 

func (scaleSet *ScaleSet) Nodes() ([]cloudprovider.Instance, error)

Nodes returns a list of all nodes that belong to this node group.

func (*ScaleSet) SetScaleSetSize 

func (scaleSet *ScaleSet) SetScaleSetSize(size int64) error

SetScaleSetSize sets ScaleSet size.

func (*ScaleSet) String 

func (m *ScaleSet) String() string

String is represented by calling GetKey()

func (*ScaleSet) TargetSize 

func (scaleSet *ScaleSet) TargetSize() (int, error)

TargetSize returns the current TARGET size of the node group. It is possible that the number is different from the number of nodes registered in Kubernetes.

func (*ScaleSet) TemplateNodeInfo 

func (scaleSet *ScaleSet) TemplateNodeInfo() (*schedulerframework.NodeInfo, error)

TemplateNodeInfo returns a node template for this scale set.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.