keights

README

keights

CloudFormation based automation for Kubernetes.

About

Keights has the following features:

Deploys in AWS using only CloudFormation and a purpose built AMI.

You don't need anything except CloudFormation to get started. The only things to download are three CloudFormation templates.

Though not required, an Ansible role is offered to deploy everything in one shot. An additional script is provided which automatically configures your kubeconfig.

One caveat is that if you are not deploying in us-east-1, you will have to copy some CloudFormation custom Lambdas from the cloudboss-public S3 bucket to a bucket in your region, as Lambdas won't deploy from a bucket outside of their region.

HA out of the box.

Keights deploys with either one or three masters. A master can be terminated and the etcd storage volume will be reattached to its replacement, allowing for rolling updates of the masters using a CloudFormation stack update.

Stays as vanilla as possible

kubeadm is used to handle most of the configuration. A command line utility handles a few extra tasks which are started as systemd services.

The addon manager is also run as a static pod on the masters, which is used to deploy the dashboard and the network, currently kube-router.

Does not manage your VPC

If you are like many companies, you don't want Kubernetes automation to build your VPC. You already have standards and policies to enforce how those are built. Keights leaves it up to you to manage your own network.

If you want Kubernetes to manage load balancers, you will need to tag subnets to let it know which ones to use, but again that is left up to you.

Does not require an internet gateway in your VPC

Keights assumes you set up your VPC the way you want it, and will not throw errors if it does not find an internet gateway. You know how to get the traffic where you want it to go, whether that means using DirectConnect, a VPN, a proxy, or some other means, and keights isn't going to stand in your way.

Is modifiable

If keights doesn't quite work for you, you can modify the CloudFormation templates or build your own AMI. It doesn't try to be all things to all people, but it does try to accomodate by not being a black box.

Usage

Note: in this guide, all commands to be run from the shell are shown preceded by > to indicate the shell prompt.

Keights has a git branch for each supported version of Kubernetes. For example, for Kubernetes 1.10.x, the branch is v1.10. Releases are numbered according to the version of Kubernetes with an incrementing suffix appended for each release supporting that Kubernetes version. For Kubernetes 1.10.0, this would be v1.10.0-1 or v1.10.0-2, for example.

The master branch follows the latest release branch, so if you are deploying the latest release, proceed. Otherwise, do a git checkout of the release branch or desired release tag.

> git checkout v1.9

Deploy with plain ol' CloudFormation

The stack/cloudformation directory contains three CloudFormation templates: common.yml, master.yml, and node.yml. The stacks must be deployed in that order.

common.yml

This contains security groups and IAM roles for the cluster.

Some of the outputs of the stack will be needed to pass to master.yml and node.yml.

These are: LoadBalancerSecurityGroup, MasterSecurityGroup, NodeSecurityGroup, MasterInstanceProfile, NodeInstanceProfile, and LambdaRoleArn.

master.yml

This contains the master stack. It will require as inputs the LoadBalancerSecurityGroup, MasterSecurityGroup, MasterInstanceProfile, and LambdaRoleArn outputs from the common.yml stack.

The MasterSecurityGroup output from common.yml should go into the MasterSecurityGroups parameter, though you may add additional groups to the list.

node.yml

This contains the node stack. You can deploy as many of these as you like, giving each one its own parameters for the node labels. It will require as inputs the NodeSecurityGroup and NodeInstanceProfile outputs from the common.yml stack, and the LoadBalancerDnsName output from the master.yml stack.

The NodeSecurityGroup output from common.yml should go into the NodeSecurityGroups parameter, though you may add additional groups to the list.

Deploy with Ansible

The stack/ansible directory contains an Ansible role keights-stack which deploys all CloudFormation stacks together, passing the outputs of one stack as inputs to another as needed. The role is published to GitHub on each keights release.

A deploy script is included in stack/ansible which creates a Python virtualenv, installs Ansible into it, uses ansible-galaxy to download the Ansible role, and then runs ansible-playbook. You need either Python 3, or Python 2 with the virtualenv command, installed and on your PATH. Python 3 now includes virtual environments using the venv standard library module.

The directory stack/ansible/example contains the recommended layout, structured according to the expectations of the deploy script.

To build a cluster, create a directory with the same layout as the example directory. All of the identifiers in example/vars.yml are fake, so edit the file to include a real VPC ID, subnet IDs, and so on. In requirements.yml, the version of the Ansible role should match the keights version.

Choose a name for the cluster. For our purpose, it is legbegbe.

You also need AWS credentials and region to be in scope. You can do this with environment variables or a credentials file.

The cluster name must be set as an environment variable called CLUSTER.

Now run deploy from the directory you created:

> export AWS_DEFAULT_REGION=us-east-1
> export AWS_PROFILE=keights
> export CLUSTER=legbegbe
> /path/to/deploy

The deploy scripts passes any additional arguments on to ansible-playbook, so for example you can increase the logging by passing -vvv to deploy.

In a few minutes, your cluster should be ready.

Keeping up to date

Keep your cluster up to date by modifying any of the files in your directory and rerunning deploy. For example, you may update the Kubernetes version by modifying the version of the keights-stack role in requirements.yml when there is a new keights release. keights strives to be able to make this transition smoothly, however always run such updates on a test cluster before running on a live system.

It is highly recommended to check your cluster's Ansible directory into source control, and let a CI/CD tool run deploy when the source changes. The Ansible role is idempotent, and there should be no changes unless the source changes. It is important to keep all versions pinned in requirements.txt and requirements.yml, to avoid unforeseen side effects.

Connecting to the cluster

In the stack directory is a script called kubesetup. It sets up ~/.kube/config for your cluster. To use it, you need both kubectl and the AWS CLI to be installed and on your PATH, and the AWS region and credentials must be in scope.

Run kubesetup with the cluster name as its only argument:

> export AWS_DEFAULT_REGION=us-east-1
> export AWS_PROFILE=keights

> ./kubesetup legbegbe

Now you can use kubectl to connect to the cluster.

> kubectl get no

AMI

A Debian AMI is created for each release containing the installed keights package and all dependencies, including required docker images.

The software should in theory run on any systemd based Linux distribution, such as RHEL, though it is currently only tested on Debian.