aws

package
v0.0.0-...-503a17f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 19, 2018 License: Apache-2.0 Imports: 27 Imported by: 0

README

Cluster Autoscaler on AWS

The cluster autoscaler on AWS scales worker nodes within any specified autoscaling group. It will run as a Deployment in your cluster. This README will go over some of the necessary steps required to get the cluster autoscaler up and running.

Kubernetes Version

Cluster autoscaler must run on v1.3.0 or greater.

Permissions

The worker running the cluster autoscaler will need access to certain resources and actions.

A minimum IAM policy would look like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

If you'd like to auto-discover node groups by specifying the --node-group-auto-discover flag, a DescribeTags permission is also required:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

AWS supports ARNs for autoscaling groups. More information here.

Deployment Specification

1 ASG Setup (min: 1, max: 10, ASG Name: k8s-worker-asg-1)
kubectl apply -f examples/cluster-autoscaler-one-asg.yaml
Multiple ASG Setup
kubectl apply -f examples/cluster-autoscaler-multi-asg.yaml
Master Node Setup

To run a CA pod in master node - CA deployment should tolerate the master taint and nodeSelector should be used to schedule the pods in master node.

kubectl apply -f examples/cluster-autoscaler-run-on-master.yaml
Auto-Discovery Setup

To run a cluster-autoscaler which auto-discovers ASGs with nodes use the --node-group-auto-discovery flag. For example, --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME> will find the ASGs where those tag keys exist. It does not matter what value the tags have.

Note that:

  • It is recommended to use a second tag like k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME> when k8s.io/cluster-autoscaler/enabled is used across many clusters to prevent ASGs from different clusters recognized as the node groups
  • There are no --nodes flags passed to cluster-autoscaler because the node groups are automatically discovered by tags
  • No min/max values are provided when using Auto-Discovery, cluster-autoscaler will respect the current min and max values of the ASG being targeted, and it will adjust only the "desired" value.
kubectl apply -f examples/cluster-autoscaler-autodiscover.yaml

Scaling a node group to 0

From CA 0.6.1 - it is possible to scale a node group to 0 (and obviously from 0), assuming that all scale-down conditions are met.

If you are using nodeSelector you need to tag the ASG with a node-template key "k8s.io/cluster-autoscaler/node-template/label/" and "k8s.io/cluster-autoscaler/node-template/taint/" if you are using taints.

For example for a node label of foo=bar you would tag the ASG with:

{
    "ResourceType": "auto-scaling-group",
    "ResourceId": "foo.example.com",
    "PropagateAtLaunch": true,
    "Value": "bar",
    "Key": "k8s.io/cluster-autoscaler/node-template/label/foo"
}

And for a taint of "dedicated": "foo:NoSchedule" you would tag the ASG with:

{
    "ResourceType": "auto-scaling-group",
    "ResourceId": "foo.example.com",
    "PropagateAtLaunch": true,
    "Value": "foo:NoSchedule",
    "Key": "k8s.io/cluster-autoscaler/node-template/taint/dedicated"
}

If you'd like to scale node groups from 0, an autoscaling:DescribeLaunchConfigurations or ec2:DescribeLaunchTemplateVersions permission is required depending on if you made your ASG with Launch Configuration or Launch Template:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeTags",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*"
        }
    ]
}

Common Notes and Gotchas:

  • The /etc/ssl/certs/ca-certificates.crt should exist by default on your ec2 instance. If you use Amazon Linux 2, use /etc/ssl/certs/ca-bundle.crt instead.
  • Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. For more information, see https://github.com/kubernetes/contrib/pull/1552#discussion_r75532949.
  • By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the --skip-nodes-with-system-pods=false flag.
  • By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the --scale-down-delay-after-add, --scale-down-delay-after-delete, and --scale-down-delay-after-failure flag. E.g. --scale-down-delay-after-add=5m to decrease the scale down delay to 5 minutes after a node has been added.
  • If you're running multiple ASGs, the --expander flag supports three options: random, most-pods and least-waste. random will expand a random ASG on scale up. most-pods will scale up the ASG that will scheduable the most amount of pods. least-waste will expand the ASG that will waste the least amount of CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back to random.

Documentation

Index

Constants

View Source 
const (
	// ProviderName is the cloud provider name for AWS
	ProviderName = "aws"
)

Variables

View Source 
var InstanceTypes = map[string]*instanceType{}/* 164 elements not displayed */

InstanceTypes is a map of ec2 resources

Functions

func BuildAWS 

func BuildAWS(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter) cloudprovider.CloudProvider

BuildAWS builds AWS cloud provider, manager etc.

func BuildAwsCloudProvider 

func BuildAwsCloudProvider(awsManager *AwsManager, resourceLimiter *cloudprovider.ResourceLimiter) (cloudprovider.CloudProvider, error)

BuildAwsCloudProvider builds CloudProvider implementation for AWS.

Types

type AwsInstanceRef 

type AwsInstanceRef struct {
	ProviderID string
	Name       string
}

AwsInstanceRef contains a reference to an instance in the AWS world.

func AwsRefFromProviderId 

func AwsRefFromProviderId(id string) (*AwsInstanceRef, error)

AwsRefFromProviderId creates InstanceConfig object from provider id which must be in format: aws:///zone/name

type AwsManager 

type AwsManager struct {
	// contains filtered or unexported fields
}

AwsManager is handles aws communication and data caching.

func CreateAwsManager 

func CreateAwsManager(configReader io.Reader, discoveryOpts cloudprovider.NodeGroupDiscoveryOptions) (*AwsManager, error)

CreateAwsManager constructs awsManager object.

func (*AwsManager) Cleanup 

func (m *AwsManager) Cleanup()

Cleanup the ASG cache.

func (*AwsManager) DeleteInstances 

func (m *AwsManager) DeleteInstances(instances []*AwsInstanceRef) error

DeleteInstances deletes the given instances. All instances must be controlled by the same ASG.

func (*AwsManager) GetAsgForInstance 

func (m *AwsManager) GetAsgForInstance(instance AwsInstanceRef) *asg

GetAsgForInstance returns AsgConfig of the given Instance

func (*AwsManager) GetAsgNodes 

func (m *AwsManager) GetAsgNodes(ref AwsRef) ([]AwsInstanceRef, error)

GetAsgNodes returns Asg nodes.

func (*AwsManager) Refresh 

func (m *AwsManager) Refresh() error

Refresh is called before every main loop and can be used to dynamically update cloud provider state. In particular the list of node groups returned by NodeGroups can change as a result of CloudProvider.Refresh().

func (*AwsManager) SetAsgSize 

func (m *AwsManager) SetAsgSize(asg *asg, size int) error

SetAsgSize sets ASG size.

type AwsNodeGroup 

type AwsNodeGroup struct {
	// contains filtered or unexported fields
}

AwsNodeGroup implements NodeGroup interface.

func (*AwsNodeGroup) Autoprovisioned 

func (ng *AwsNodeGroup) Autoprovisioned() bool

Autoprovisioned returns true if the node group is autoprovisioned.

func (*AwsNodeGroup) Belongs 

func (ng *AwsNodeGroup) Belongs(node *apiv1.Node) (bool, error)

Belongs returns true if the given node belongs to the NodeGroup.

func (*AwsNodeGroup) Create 

func (ng *AwsNodeGroup) Create() (cloudprovider.NodeGroup, error)

Create creates the node group on the cloud provider side.

func (*AwsNodeGroup) Debug 

func (ng *AwsNodeGroup) Debug() string

Debug returns a debug string for the Asg.

func (*AwsNodeGroup) DecreaseTargetSize 

func (ng *AwsNodeGroup) DecreaseTargetSize(delta int) error

DecreaseTargetSize decreases the target size of the node group. This function doesn't permit to delete any existing node and can be used only to reduce the request for new nodes that have not been yet fulfilled. Delta should be negative. It is assumed that cloud provider will not delete the existing nodes if the size when there is an option to just decrease the target.

func (*AwsNodeGroup) Delete 

func (ng *AwsNodeGroup) Delete() error

Delete deletes the node group on the cloud provider side. This will be executed only for autoprovisioned node groups, once their size drops to 0.

func (*AwsNodeGroup) DeleteNodes 

func (ng *AwsNodeGroup) DeleteNodes(nodes []*apiv1.Node) error

DeleteNodes deletes the nodes from the group.

func (*AwsNodeGroup) Exist 

func (ng *AwsNodeGroup) Exist() bool

Exist checks if the node group really exists on the cloud provider side. Allows to tell the theoretical node group from the real one.

func (*AwsNodeGroup) Id 

func (ng *AwsNodeGroup) Id() string

Id returns asg id.

func (*AwsNodeGroup) IncreaseSize 

func (ng *AwsNodeGroup) IncreaseSize(delta int) error

IncreaseSize increases Asg size

func (*AwsNodeGroup) MaxSize 

func (ng *AwsNodeGroup) MaxSize() int

MaxSize returns maximum size of the node group.

func (*AwsNodeGroup) MinSize 

func (ng *AwsNodeGroup) MinSize() int

MinSize returns minimum size of the node group.

func (*AwsNodeGroup) Nodes 

func (ng *AwsNodeGroup) Nodes() ([]string, error)

Nodes returns a list of all nodes that belong to this node group.

func (*AwsNodeGroup) TargetSize 

func (ng *AwsNodeGroup) TargetSize() (int, error)

TargetSize returns the current TARGET size of the node group. It is possible that the number is different from the number of nodes registered in Kubernetes.

func (*AwsNodeGroup) TemplateNodeInfo 

func (ng *AwsNodeGroup) TemplateNodeInfo() (*schedulercache.NodeInfo, error)

TemplateNodeInfo returns a node template for this node group.

type AwsRef 

type AwsRef struct {
	Name string
}

AwsRef contains a reference to some entity in AWS world.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.