README ¶
Kubernetes Cloud Controller Manager for Yandex.Cloud
Overview
yandex-cloud-controller-manager
is the Kubernetes Cloud Controller Manager (CCM) implementation for Yandex.Cloud.
It allows you to leverage many of the cloud provider features offered by Yandex.Cloud on your Kubernetes clusters.
Read more about Kubernetes CCM here.
Currently yandex-cloud-controller-manager
implements:
NodeController
- responsible for updating kubernetes nodes with cloud provider specific labels and addresses and deleting kubernetes nodes that were deleted on your cloud.ServiceController
- responsible for creating LoadBalancers when a service ofType: LoadBalancer
is created in Kubernetes.RouteController
- responsible for creating routes to allow networking between Pods.
Work In Progress
This project is currently under active development. Use at your own risk! Contributions welcome!
Getting Started
Requirements
At the current state of Kubernetes, running Cloud Controller Manager (CCM) requires a few things. Please read through the requirements carefully as they are critical to running CCM on a Kubernetes cluster on Yandex.Cloud.
Version
Kubernetes 1.16+
Cloud resources
- All Kubernetes nodes MUST be located in the same
Folder
. For more details about folders - refer to official documentation - Kubernetes node names MUST match the VM name.
By default, the
kubelet
will name nodes based on the node hostname. On Yandex.Cloud, node hostname is set based on the VM name. So, it is important that the node name on Kubernetes matches corresponding VM name, otherwise CCM will not be able to find corresponding cloud resources.
Cluster configuration
kubelet
MUST run with--cloud-provider=external
. This is to ensure that thekubelet
is aware that it must be initialized by the CCM before it is scheduled any work.kube-apiserver
andkube-controller-manager
MUST NOT set the flag--cloud-provider
which will default them to use no cloud provider natively.
WARNING: setting --cloud-provider=external
will taint all nodes in a cluster with node.cloudprovider.kubernetes.io/uninitialized
.
It is the responsibility of CCM to untaint those nodes once it has finished initializing them.
This means that most pods will be left unschedulable until the CCM is running.
Deployment
Caution: Yandex.Cloud's documentation refers to Network or VPC interchangeably. If a Subnet ID is required in the configuration, it will be explicitly stated.
Authentication and Configuration
The yandex-cloud-controller-manager
requires a Service Account Json, Folder ID and a Cloud-unique cluster Name stored in the following environment variables:
YANDEX_CLOUD_SERVICE_ACCOUNT_JSON
YANDEX_CLOUD_FOLDER_ID
YANDEX_CLUSTER_NAME
The default manifest is configured to set these environment variables from a secret named yandex-cloud
:
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: yandex-cloud-controller-manager
name: yandex-cloud
namespace: kube-system
stringData:
service-account-json: |
{
"id": "ajesh3orip69r0vctpf5",
"service_account_id": "aje3qbblkdf2u2avn4n7",
"created_at": "2020-01-20T07:43:49Z",
"key_algorithm": "RSA_2048",
"public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAozah4aqf9xkMLRRtNJjz\nZ+xooLV6GLGaHbkl3796r2zWbgm1aNU3xILGeStdTM5XbB651OAfq7YyHoDSiZkj\nBP6W2ZYNO/WjK9N13rWhtFjNiDulLh3LAU47qNy75OsC3SjW58MVHPNriIgB0HLA\nHRE6cguUJtUcKWqOKhoKQVJxXLOhsdjmEEdnHtd9ro3UCcPM7r6fc+MmkCaZgTNl\nkItkDDxodTIqj3Go2EiEyO2DaMye+GqTzSNJOYaHFH4DYhCCgE1/SCY356nER2qH\nymbAGkD2fAp2eGoCEM67AgqrWFEm/yI+FlIvcrn7wC6+NfWUVqPb+N4wuiehCRBO\n8wIDAQAB\n-----END PUBLIC KEY-----\n",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCjNqHhqp/3GQwt\nFG00mPNn7GigtXoYsZoduSXfv3qvbNZuCbVo1TfEgsZ5K11MzldsHrnU4B+rtjIe\ngNKJmSME/pbZlg079aMr03XetaG0WM2IO6UuHcsBTjuo3Lvk6wLdKNbnwxUc82uI\niAHQcsAdETpyC5Qm1Rwpao4qGgpBUnFcs6Gx2OYQR2ce132ujdQJw8zuvp9z4yaQ\nJpmBM2WQi2QMPGh1MiqPcajYSITI7YNozJ74apPNI0k5hocUfgNiEIKATX9IJjfn\nqcRHaofKZsAaQPZ8CnZ4agIQzrsCCqtYUSb/Ij4WUi9yufvALr419ZRWo9v43jC6\nJ6EJEE7zAgMBAAECggEAF0hi3XNesHw1PXUNgxRSnL+fyVU6Hq2vQ5A28+03zjCj\ngj0GUPchpnnVYFGsVJmW5QiZD+INAozSJ4HPBuv+j+bVlCKQrr4C0eyvgt68O6Lz\nZvzDOonrfLsxTYx3jVdtKCl8RsGQkHm1HFvyjk7gUwUzJjO6pbN++fWGZEEkt16W\nFHaGldz2MvZKOwQwj0WUcjF4X8PWTvJ0Ar1i5XoAm35GN+2ziwJKcNt+DsJ3N6MW\ngAkivYE8f44T3fQFg5M1RC6v2Jp2lmtVRxYRI0rcie+HyxJVcHgWTZPdTkwGWKDD\nnRU2OTJoZCJf3BunFtB1P8W+GlmLFdBjTppFhgqI6QKBgQDO61fX49qVRDmORYor\nVWh1tZkw546llwkNqLAe1QoLhqjHGctUs3lOczDqI82PwGKb423JdgKmr9nzrCZW\nq5a/BulHdsGsvkSBGK091fhQYRQnaTF7bnXoyVI8qUerGiV/a8/7W1SM+WIJayZ3\nr5Z9xV9LH/Wy7uWW7Xr2LvP97wKBgQDJ7VuYZVJ17hPIqEaR3P1Jvka+RvusWTPw\n1o6O935tW28Q/S2J661Dv92mTNmll/beyFS1ZkHdQ/1c/92Pr1bM/A4CrQoNDvad\nhd2MnyzVYyHc4p6Yd6VmZisbPpTfa7ZJMzYUN27nj+yPxJyZ/EoLlcXaXcPV3kFo\nZsubNT0DPQKBgQCz/nLmgPWWnMd4ZDOB6IS6yCKfMP6cOtsMP64c0/Mt/ZB5yY1f\ne9PNE1T8h/J71r2wn1DUS8yYlSYB2sFq6U5zk55/pOVq0AQlTIL+5E9iFGCEu/Po\nTDlTKzVXQWXviAoQYoeEPnk5PII0cToAKQS/GV8AqaeAZGHhPWmWF1f1jwKBgHQx\nJ6aejv+bGjk5Uzo1rm3TloOA9uqqfa/U1j0//rjQhy2AccbOHWpBqjo6OHcH5Z82\nKUAkcjvvFoiAFq7KVykm1K0HgyQWeyQTVnPHWBYFsAOZR2c2Wa99lMpdjW6uXTrr\nw++IIkIO2DG2EeKtgLH/4dSQZdLXzE1V8U0DKnOFAoGAQNCBpnE1WHR9H5APr5SF\nuD35dTm3O2OvczlbB0MUhx8R7qPpvLwA5HRSIKAKxobUbGpdgCy6WuncRWg+TjaD\n8zlwZvG2+vtntCFPcIT6ZpGH6k9ppXOPJBxaHZRHJSoilGhF1KvrmY8T5WxTVuyM\nnmypFU40LHcTmvw/a6JY+BM=\n-----END PRIVATE KEY-----\n"
}
folder-id: "b1g4c2a3g6vkffp3qacq"
EOF
Installation - with RBAC
kubectl apply -f manifests/yandex-cloud-controller-manager-rbac.yaml
kubectl apply -f manifests/yandex-cloud-controller-manager.yaml
Installation - without RBAC
kubectl apply -f manifests/yandex-cloud-controller-manager.yaml
NOTE: the deployments in manifests
folder are meant to serve as an example.
They will work in a majority of cases but may not work out of the box for your cluster.
Subsystem-specific information
Node Controller
YANDEX_CLOUD_INTERNAL_NETWORK_IDS
– comma separated list of NetworkIDs. Will be used to select InternalIPs when scanning an Yandex Instance and populating the corresponding Kubernetes Node.- Optional.
- If present, we iterate over all Instance's interfaces and select networkID-matching addresses.
- If not present, we use an IP address from the first interface as an InternalIP.
YANDEX_CLOUD_EXTERNAL_NETWORK_IDS
– comma separated list of NetworkIDs. Will be used to select ExternalIPs when scanning an Yandex Instance and populating the corresponding Kubernetes Node.- Optional.
- If present, we iterate over all Instance's interfaces and select networkID-matching private addresses.
- If not present, we use public address from the first interface that has one-to-one NAT enabled, or none at all.
Service Controller
Due to Yandex.Cloud's TargetGroup's inability to create duplicate (same SubnetID and IP address) Targets, this CCM takes all Nodes in the clusters, finds its Yandex.Cloud Instance counterpart via the API, scans the Instance's Interfaces and determines all possible sets of NetworkIDs that exist on these instances. After doing so, it creates multiple TargetGroups that are named after aforementioned NetworkIDs and include Targets from said Networks.
Due to API limitations, only one subnet from each zone must be present in each NetworkID present on Instance's network interfaces.
YANDEX_CLOUD_DEFAULT_LB_TARGET_GROUP_NETWORK_ID
– default NetworkID to use for TargetGroup for created NetworkLoadBalancers.- Mandatory.
YANDEX_CLOUD_DEFAULT_LB_LISTENER_SUBNET_ID
– default SubnetID to use for created NetworkLoadBalancers' listeners.- Caution! All newly created NLBs will be INTERNAL. This can be overriden via
yandex.cpi.flant.com/loadbalancer-external
Service annotation.
- Caution! All newly created NLBs will be INTERNAL. This can be overriden via
yandex.cpi.flant.com/target-group-network-id
– overrideYANDEX_CLOUD_DEFAULT_LB_TARGET_GROUP_NETWORK_ID
on a per-service basis.yandex.cpi.flant.com/listener-subnet-id
– default SubnetID to use for Listeners in created NetworkLoadBalancers. NetworkLoadBalancers will be INTERNAL.yandex.cpi.flant.com/listener-address-ipv4
– select pre-defined IPv4 address. Works both on internal and external NetworkLoadBalancers.yandex.cpi.flant.com/loadbalancer-external
– overrideYANDEX_CLOUD_DEFAULT_LB_LISTENER_SUBNET_ID
per-service.yandex.cpi.flant.com/target-group-name-prefix
- set target group for LB to target group with nameyandex.cpi.flant.com/target-group-name-prefix
annotation value + yandex cluster name +YANDEX_CLOUD_DEFAULT_LB_TARGET_GROUP_NETWORK_ID
.yandex.cpi.flant.com/healthcheck-interval-seconds
- healthcheck interval(default 2).yandex.cpi.flant.com/healthcheck-timeout-seconds
- healthcheck timeout(default 1).yandex.cpi.flant.com/healthcheck-unhealthy-threshold
- healthcheck unhealthy threshold(default 2).yandex.cpi.flant.com/healthcheck-healthy-threshold
- healthcheck healthy threshold(default 2).
yandex.cpi.flant.com/target-group-name-prefix
- set node to the non-default target group add this annotation to the node. Yandex CCM creates new target groups with nameyandex.cpi.flant.com/target-group-name-prefix
annotation value + yandex cluster name + network id of instance interfaces.
Warning
-
If masters are created with their own target groups, then you need to attach the
node.kubernetes.io/exclude-from-external-load-balancers: ""
label on them so that the controller does not try to add the master to a new target group for balancers -
If you want to add a balancer for the masters yourself, so that ycc could also hang its own balancers on the masters, create a target group in advance by the name mask
${CLUSTER-NAME}${VPC.ID}
Development
The yandex-cloud-controller-manager
is written in Google's Go programming language.
Currently, it is developed and tested on Go 1.13.6.
If you haven't set up a Go development environment yet, please follow these instructions.
Download Source
$ go get -u github.com/deckhouse/yandex-cloud-controller-manager
$ cd $(go env GOPATH)/src/github.com/deckhouse/yandex-cloud-controller-manager
Dependency management
yandex-cloud-controller-manager
uses Go modules to manage dependencies.
Build Binary
To build yandex-cloud-controller-manager
binary, run:
$ make build
Building Docker images
To build Docker image, use the following make target:
$ DOCKER_TAG=dev make docker-build