cherryservers

package
v0.0.0-...-29ce5d4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 27, 2024 License: Apache-2.0 Imports: 32 Imported by: 0

README

Cluster Autoscaler for Cherry Servers

The cluster autoscaler for Cherry Servers worker nodes performs autoscaling within any specified nodepools. It will run as a Deployment in your cluster. The nodepools are specified using tags on Cherry Servers.

This README will go over some of the necessary steps required to get the cluster autoscaler up and running.

Permissions and credentials

The autoscaler needs a ServiceAccount with permissions for Kubernetes and requires credentials, specifically API tokens, for interacting with Cherry Servers.

An example ServiceAccount is given in examples/cluster-autoscaler-svcaccount.yaml.

The credentials for authenticating with Cherry Servers are stored in a secret and provided as an env var to the container. examples/cluster-autoscaler-secret In the above file you can modify the following fields:

Secret Key Value
cluster-autoscaler-cherry authtoken Your Cherry Servers API token. It must be base64 encoded.
cluster-autoscaler-cloud-config Global/project-id Your Cherry Servers project id
cluster-autoscaler-cloud-config Global/api-server The ip:port for you cluster's k8s api (e.g. K8S_MASTER_PUBLIC_IP:6443)
cluster-autoscaler-cloud-config Global/region The Cherry Servers region slug for the servers in your nodepool (eg: eu_nord_1)
cluster-autoscaler-cloud-config Global/plan The Cherry Servers plan slug for new nodes in the nodepool (eg: e5_1620v4)
cluster-autoscaler-cloud-config Global/os The OS image slug to use for new nodes, e.g. ubuntu_18_04. If you change this also update cloudinit.
cluster-autoscaler-cloud-config Global/cloudinit The base64 encoded user data submitted when provisioning servers. In the example file, the default value has been tested with Ubuntu 18.04 to install Docker & kubelet and then to bootstrap the node into the cluster using kubeadm. The kubeadm, kubelet, kubectl are pinned to version 1.17.4. For a different base OS or bootstrap method, this needs to be customized accordingly. It will use go templates to inject runtime information; see below.
cluster-autoscaler-cloud-config Global/hostname-pattern The pattern for the names of new Cherry Servers servers (default: "k8s-{{.ClusterName}}-{{.NodeGroup}}-{{.RandString8}}" )
cluster-autoscaler-cloud-config Global/os-partition-size The OS partition size in gigabytes for new nodes in the nodepool (eg: 60, default: none)

You can always update the secret with more nodepool definitions (with different plans etc.) as shown in the example, but you should always provide a default nodepool configuration.

The userdata use the following fields to inject runtime information into userdata:

  • BootstrapTokenID: Kubernetes bootstrap token ID, 6 alphanumeric characters, e.g. nf8atf
  • BootstrapTokenSecret: Kubernetes bootstrap token secret, 16 alphanumeric characters, e.g. kwfs1t15pjmhk7n4
  • APIServerEndpoint: endpoint to connect to the Kubernetes server
  • NodeGroup: name of the nodegroup to which the newly deployed node belongs.

For example:

kubeadm join --discovery-token-unsafe-skip-ca-verification --token {{.BootstrapTokenID}}.{{.BootstrapTokenSecret}} {{.APIServerEndpoint}}

Configure nodepool and cluster names using Cherry Servers tags

The Cherry Servers API does not yet have native support for groups or pools of servers. So we use tags to specify them. Each Cherry Servers server that's a member of the "cluster1" cluster should have the tag k8s-cluster-cluster1. The servers that are members of the "pool1" nodepool should also have the tag k8s-nodepool-pool1. Once you have a Kubernetes cluster running on Cherry Servers, use the Cherry Servers Portal, API or CLI to tag the nodes accordingly.

Autoscaler deployment

The yaml files in examples can be used. You will need to change several of the files to match your cluster:

Argument Usage
--cluster-name The name of your Kubernetes cluster. It should correspond to the tags that have been applied to the nodes.
--nodes Of the form min:max:NodepoolName. For multiple nodepools you can add the same argument multiple times. E.g. for pool1, pool2 you would add --nodes=0:10:pool1 and --nodes=0:10:pool2. In addition, each node provisioned by the autoscaler will have a label with key: pool and with value: NodepoolName. These labels can be useful when there is a need to target specific nodepools.
--expander=random This is an optional argument which allows the cluster-autoscaler to take into account various algorithms when scaling with multiple nodepools, see expanders.

Target Specific Nodepools

In case you want to target a specific nodepool(s) for e.g. a deployment, you can add a nodeAffinity with the key pool and with value the nodepool name that you want to target. This functionality is not backwards compatible, which means that nodes provisioned with older cluster-autoscaler images won't have the key pool. But you can overcome this limitation by manually adding the correct labels. Here are some examples:

Target a nodepool with a specific name:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: pool
          operator: In
          values:
          - pool3

Target a nodepool with a specific Cherry Servers instance:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: beta.kubernetes.io/instance-type
          operator: In
          values:
          - e5_1620v4

CCM and Controller node labels

CCM

By default, autoscaler assumes that you have a recent version of Cherry Servers CCM installed in your cluster.

Notes

The autoscaler will not remove nodes which have non-default kube-system pods. This prevents the node that the autoscaler is running on from being scaled down. If you are deploying the autoscaler into a cluster which already has more than one node, it is best to deploy it onto any node which already has non-default kube-system pods, to minimise the number of nodes which cannot be removed when scaling. For this reason in the provided example the autoscaler pod has a nodeaffinity which forces it to deploy on the control plane (previously referred to as master) node.

Development

Testing

The Cherry Servers cluster-autoscaler includes a series of tests, which are executed against a mock backend server included in the package. It will not execute them against the real Cherry Servers API.

If you want to execute them against the real Cherry Servers API, set the environment variable:

CHERRY_USE_PRODUCTION_API=true
Running Locally

To run the CherryServers cluster-autoscaler locally:

  1. Save the desired cloud-config to a local file, e.g. /tmp/cloud-config. The contents of the file can be extracted from the value in examples/cluster-autoscaler-secret.yaml, secret named cluster-autoscaler-cloud-config, key cloud-config.
  2. Export the following environment variables:
    • BOOTSTRAP_TOKEN_ID: the bootstrap token ID, i.e. the leading 6 characters of the entire bootstrap token, before the .
    • BOOTSTRAP_TOKEN_SECRET: the bootstrap token secret, i.e. the trailing 16 characters of the entire bootstrap token, after the .
    • CHERRY_AUTH_TOKEN: your CherryServers authentication token
    • KUBECONFIG: a kubeconfig file with permissions to your cluster
    • CLUSTER_NAME: the name for your cluster, e.g. cluster1
    • CLOUD_CONFIG: the path to your cloud-config file, e.g. /tmp/cloud-config
  3. Run the autoscaler per the command-line below.

The command-line format is:

cluster-autoscaler --alsologtostderr --cluster-name=$CLUSTER_NAME --cloud-config=$CLOUD_CONFIG \     
  --cloud-provider=cherryservers \
  --nodes=0:10:pool1 \
  --nodes=0:10:pool2 \
  --scale-down-unneeded-time=1m0s --scale-down-delay-after-add=1m0s --scale-down-unready-time=1m0s \
  --kubeconfig=$KUBECONFIG \
  --v=2

You can set --nodes= as many times as you like. The format for each --nodes= is:

--nodes=<min>:<max>:<poolname>
  • <min> and <max> must be integers, and <max> must be greater than <min>
  • <poolname> must be a pool that exists in the cloud-config

If the poolname is not found, it will use the default pool, e.g.:

You also can make changes and run it directly, replacing the command with go run, but this must be run from the cluster-autoscaler directory, i.e. not within the specific cloudprovider implementation:

go run . --alsologtostderr --cluster-name=$CLUSTER_NAME --cloud-config=$CLOUD_CONFIG \     
  --cloud-provider=cherryservers \
  --nodes=0:10:pool1 \
  --nodes=0:10:pool2 \
  --scale-down-unneeded-time=1m0s --scale-down-delay-after-add=1m0s --scale-down-unready-time=1m0s \
  --kubeconfig=$KUBECONFIG \
  --v=2

Documentation

Index

Constants

View Source 
const (
	// ProviderName is the cloud provider name for Cherry Servers
	ProviderName = "cherryservers"
	// GPULabel is the label added to nodes with GPU resource.
	GPULabel = "cherryservers.com/gpu"
	// DefaultControllerNodeLabelKey is the label added to Master/Controller to identify as
	// master/controller node.
	DefaultControllerNodeLabelKey = "node-role.kubernetes.io/master"
	// ControllerNodeIdentifierEnv is the string for the environment variable.
	ControllerNodeIdentifierEnv = "CHERRY_CONTROLLER_NODE_IDENTIFIER_LABEL"
)

Variables

This section is empty.

Functions

func BuildCherry 

func BuildCherry(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter) cloudprovider.CloudProvider

BuildCherry is called by the autoscaler to build a Cherry Servers cloud provider.

The cherryManager is created here, and the node groups are created based on the specs provided via the command line parameters.

func BuildGenericLabels 

func BuildGenericLabels(nodegroup string, plan *Plan) map[string]string

BuildGenericLabels builds basic labels for Cherry Servers nodes

func Contains 

func Contains(a []string, x string) bool

Contains tells whether a contains x.

func Find 

func Find(a []string, x string) int

Find returns the smallest index i at which x == a[i], or len(a) if there is no such index.

Types

type AssignedTo 

type AssignedTo struct {
	ID       int     `json:"id,omitempty"`
	Name     string  `json:"name,omitempty"`
	Href     string  `json:"href,omitempty"`
	Hostname string  `json:"hostname,omitempty"`
	Image    string  `json:"image,omitempty"`
	Region   Region  `json:"region,omitempty"`
	State    string  `json:"state,omitempty"`
	Pricing  Pricing `json:"pricing,omitempty"`
}

AssignedTo assignment of a network floating IP to a server

type AttachedTo 

type AttachedTo struct {
	Href string `json:"href"`
}

AttachedTo what a resource is attached to

type AvailableRegions 

type AvailableRegions struct {
	ID         int    `json:"id,omitempty"`
	Name       string `json:"name,omitempty"`
	RegionIso2 string `json:"region_iso_2,omitempty"`
	StockQty   int    `json:"stock_qty,omitempty"`
}

AvailableRegions regions that are available to the user

type BGPRoute 

type BGPRoute struct {
	Subnet  string `json:"subnet,omitempty"`
	Active  bool   `json:"active,omitempty"`
	Router  string `json:"router,omitempty"`
	Age     string `json:"age,omitempty"`
	Updated string `json:"updated,omitempty"`
}

BGPRoute single server BGP route

type Bandwidth 

type Bandwidth struct {
	Name string `json:"name,omitempty"`
}

Bandwidth total bandwidth available

type BlockStorage 

type BlockStorage struct {
	ID            int        `json:"id"`
	Name          string     `json:"name"`
	Href          string     `json:"href"`
	Size          int        `json:"size"`
	AllowEditSize bool       `json:"allow_edit_size"`
	Unit          string     `json:"unit"`
	Description   string     `json:"description,omitempty"`
	AttachedTo    AttachedTo `json:"attached_to,omitempty"`
	VlanID        string     `json:"vlan_id"`
	VlanIP        string     `json:"vlan_ip"`
	Initiator     string     `json:"initiator"`
	DiscoveryIP   string     `json:"discovery_ip"`
}

BlockStorage cloud block storage

type CloudInitTemplateData 

type CloudInitTemplateData struct {
	BootstrapTokenID     string
	BootstrapTokenSecret string
	APIServerEndpoint    string
	NodeGroup            string
}

CloudInitTemplateData represents the variables that can be used in cloudinit templates

type ConfigFile 

type ConfigFile struct {
	DefaultNodegroupdef ConfigNodepool             `gcfg:"global"`
	Nodegroupdef        map[string]*ConfigNodepool `gcfg:"nodegroupdef"`
}

ConfigFile is used to read and store information from the cloud configuration file

type ConfigNodepool 

type ConfigNodepool struct {
	ClusterName       string   `gcfg:"cluster-name"`
	ProjectID         int      `gcfg:"project-id"`
	APIServerEndpoint string   `gcfg:"api-server-endpoint"`
	Region            string   `gcfg:"region"`
	Plan              string   `gcfg:"plan"`
	OS                string   `gcfg:"os"`
	SSHKeys           []string `gcfg:"ssh-key-ids"`
	CloudInit         string   `gcfg:"cloudinit"`
	HostnamePattern   string   `gcfg:"hostname-pattern"`
	OsPartitionSize   int      `gcfg:"os-partition-size"`
}

ConfigNodepool options only include the project-id for now

func (ConfigNodepool) IsEmpty 

func (c ConfigNodepool) IsEmpty() bool

IsEmpty determine if this is an empty config

type Cpus 

type Cpus struct {
	Count     int     `json:"count,omitempty"`
	Name      string  `json:"name,omitempty"`
	Cores     int     `json:"cores,omitempty"`
	Frequency float32 `json:"frequency,omitempty"`
	Unit      string  `json:"unit,omitempty"`
}

Cpus cpu information for a server

type CreateServer 

type CreateServer struct {
	ProjectID       int                `json:"project_id,omitempty"`
	Plan            string             `json:"plan,omitempty"`
	Hostname        string             `json:"hostname,omitempty"`
	Image           string             `json:"image,omitempty"`
	Region          string             `json:"region,omitempty"`
	SSHKeys         []int              `json:"ssh_keys"`
	IPAddresses     []string           `json:"ip_addresses"`
	UserData        string             `json:"user_data,omitempty"`
	Tags            *map[string]string `json:"tags,omitempty"`
	SpotInstance    int                `json:"spot_market,omitempty"`
	OSPartitionSize int                `json:"os_partition_size,omitempty"`
}

CreateServer represents a request to create a new Cherry Servers server. Used by createNodes

type ErrorResponse 

type ErrorResponse struct {
	Response    *http.Response
	Errors      []string `json:"errors"`
	SingleError string   `json:"error"`
}

ErrorResponse is the http response used on errors

func (*ErrorResponse) Error 

func (r *ErrorResponse) Error() string

Error implements the error interface

type HostnameTemplateData 

type HostnameTemplateData struct {
	ClusterName string
	NodeGroup   string
	RandString8 string
}

HostnameTemplateData represents the template variables used to construct host names for new nodes

type IPAddressCreateRequest 

type IPAddressCreateRequest struct {
	AddressFamily int  `json:"address_family"`
	Public        bool `json:"public"`
}

IPAddressCreateRequest represents a request to create a new IP address within a CreateServer request

type IPAddresses 

type IPAddresses struct {
	ID            string            `json:"id,omitempty"`
	Address       string            `json:"address,omitempty"`
	AddressFamily int               `json:"address_family,omitempty"`
	Cidr          string            `json:"cidr,omitempty"`
	Gateway       string            `json:"gateway,omitempty"`
	Type          string            `json:"type,omitempty"`
	Region        Region            `json:"region,omitempty"`
	RoutedTo      RoutedTo          `json:"routed_to,omitempty"`
	AssignedTo    AssignedTo        `json:"assigned_to,omitempty"`
	TargetedTo    AssignedTo        `json:"targeted_to,omitempty"`
	Project       Project           `json:"project,omitempty"`
	PtrRecord     string            `json:"ptr_record,omitempty"`
	ARecord       string            `json:"a_record,omitempty"`
	Tags          map[string]string `json:"tags,omitempty"`
	Href          string            `json:"href,omitempty"`
}

IPAddresses individual IP address

type Memory 

type Memory struct {
	Count int    `json:"count,omitempty"`
	Total int    `json:"total,omitempty"`
	Unit  string `json:"unit,omitempty"`
	Name  string `json:"name,omitempty"`
}

Memory cpu information for a server

type Nics 

type Nics struct {
	Name string `json:"name,omitempty"`
}

Nics network interface information for a server

type NodeRef 

type NodeRef struct {
	Name       string
	MachineID  string
	ProviderID string
	IPs        []string
}

NodeRef stores the name, machineID and providerID of a node.

type Plan 

type Plan struct {
	ID               int                `json:"id,omitempty"`
	Slug             string             `json:"slug,omitempty"`
	Name             string             `json:"name,omitempty"`
	Custom           bool               `json:"custom,omitempty"`
	Specs            Specs              `json:"specs,omitempty"`
	Pricing          []Pricing          `json:"pricing,omitempty"`
	AvailableRegions []AvailableRegions `json:"available_regions,omitempty"`
}

Plan a server plan

type Plans 

type Plans []Plan

Plans represents a list of Cherry Servers plans

type Pricing 

type Pricing struct {
	Price    float32 `json:"price,omitempty"`
	Taxed    bool    `json:"taxed,omitempty"`
	Currency string  `json:"currency,omitempty"`
	Unit     string  `json:"unit,omitempty"`
}

Pricing price for a specific plan

type Project 

type Project struct {
	ID   int        `json:"id,omitempty"`
	Name string     `json:"name,omitempty"`
	Bgp  ProjectBGP `json:"bgp,omitempty"`
	Href string     `json:"href,omitempty"`
}

Project a CherryServers project

type ProjectBGP 

type ProjectBGP struct {
	Enabled  bool `json:"enabled,omitempty"`
	LocalASN int  `json:"local_asn,omitempty"`
}

ProjectBGP information about BGP on an individual project

type Raid 

type Raid struct {
	Name string `json:"name,omitempty"`
}

Raid raid for block storage on a server

type Region 

type Region struct {
	ID         int       `json:"id,omitempty"`
	Slug       string    `json:"slug,omitempty"`
	Name       string    `json:"name,omitempty"`
	RegionIso2 string    `json:"region_iso_2,omitempty"`
	BGP        RegionBGP `json:"bgp,omitempty"`
	Href       string    `json:"href,omitempty"`
}

Region a CherryServers region

type RegionBGP 

type RegionBGP struct {
	Hosts []string `json:"hosts,omitempty"`
	Asn   int      `json:"asn,omitempty"`
}

RegionBGP information about BGP in a region

type RoutedTo 

type RoutedTo struct {
	ID            string `json:"id,omitempty"`
	Address       string `json:"address,omitempty"`
	AddressFamily int    `json:"address_family,omitempty"`
	Cidr          string `json:"cidr,omitempty"`
	Gateway       string `json:"gateway,omitempty"`
	Type          string `json:"type,omitempty"`
	Region        Region `json:"region,omitempty"`
}

RoutedTo routing of a floating IP to an underlying IP

type SSHKeys 

type SSHKeys struct {
	ID          int    `json:"id,omitempty"`
	Label       string `json:"label,omitempty"`
	Key         string `json:"key,omitempty"`
	Fingerprint string `json:"fingerprint,omitempty"`
	Updated     string `json:"updated,omitempty"`
	Created     string `json:"created,omitempty"`
	Href        string `json:"href,omitempty"`
}

SSHKeys an ssh key

type Server 

type Server struct {
	ID               int               `json:"id,omitempty"`
	Name             string            `json:"name,omitempty"`
	Href             string            `json:"href,omitempty"`
	Hostname         string            `json:"hostname,omitempty"`
	Image            string            `json:"image,omitempty"`
	SpotInstance     bool              `json:"spot_instance"`
	BGP              ServerBGP         `json:"bgp,omitempty"`
	Project          Project           `json:"project,omitempty"`
	Region           Region            `json:"region,omitempty"`
	State            string            `json:"state,omitempty"`
	Plan             Plan              `json:"plan,omitempty"`
	AvailableRegions AvailableRegions  `json:"availableregions,omitempty"`
	Pricing          Pricing           `json:"pricing,omitempty"`
	IPAddresses      []IPAddresses     `json:"ip_addresses,omitempty"`
	SSHKeys          []SSHKeys         `json:"ssh_keys,omitempty"`
	Tags             map[string]string `json:"tags,omitempty"`
	Storage          BlockStorage      `json:"storage,omitempty"`
	Created          string            `json:"created_at,omitempty"`
	TerminationDate  string            `json:"termination_date,omitempty"`
}

Server represents a Cherry Servers server

type ServerBGP 

type ServerBGP struct {
	Enabled   bool       `json:"enabled"`
	Available bool       `json:"available,omitempty"`
	Status    string     `json:"status,omitempty"`
	Routers   int        `json:"routers,omitempty"`
	Connected int        `json:"connected,omitempty"`
	Limit     int        `json:"limit,omitempty"`
	Active    int        `json:"active,omitempty"`
	Routes    []BGPRoute `json:"routes,omitempty"`
	Updated   string     `json:"updated,omitempty"`
}

ServerBGP status of BGP on a server

type Specs 

type Specs struct {
	Cpus      Cpus      `json:"cpus,omitempty"`
	Memory    Memory    `json:"memory,omitempty"`
	Storage   []Storage `json:"storage,omitempty"`
	Raid      Raid      `json:"raid,omitempty"`
	Nics      Nics      `json:"nics,omitempty"`
	Bandwidth Bandwidth `json:"bandwidth,omitempty"`
}

Specs aggregated specs for a server

type Storage 

type Storage struct {
	Count int     `json:"count,omitempty"`
	Name  string  `json:"name,omitempty"`
	Size  float32 `json:"size,omitempty"`
	Unit  string  `json:"unit,omitempty"`
}

Storage amount of storage

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.