keights
CloudFormation based automation for Kubernetes.
About
Keights has the following features:
You don't need anything except CloudFormation to get started. The only things to download are three CloudFormation templates.
Though not required, an Ansible role is offered to deploy everything in one shot. An additional script is provided which automatically configures your kubeconfig.
One caveat is that if you are not deploying in us-east-1
, you will have to copy some CloudFormation custom Lambdas from the cloudboss-public
S3 bucket to a bucket in your region, as Lambdas won't deploy from a bucket outside of their region.
HA out of the box.
Keights deploys with either one or three masters. A master can be terminated and the etcd storage volume will be reattached to its replacement, allowing for rolling updates of the masters using a CloudFormation stack update.
Stays as vanilla as possible
kubeadm
is used to handle most of the configuration. A command line utility handles a few extra tasks which are started as systemd services.
The addon manager is also run as a static pod on the masters, which is used to deploy the dashboard and the network, currently kube-router.
Does not manage your VPC
If you are like many companies, you don't want Kubernetes automation to build your VPC. You already have standards and policies to enforce how those are built. Keights leaves it up to you to manage your own network.
If you want Kubernetes to manage load balancers, you will need to tag subnets to let it know which ones to use, but again that is left up to you.
Does not require an internet gateway in your VPC
Keights assumes you set up your VPC the way you want it, and will not throw errors if it does not find an internet gateway. You know how to get the traffic where you want it to go, whether that means using DirectConnect, a VPN, a proxy, or some other means, and keights isn't going to stand in your way.
Is modifiable
If keights doesn't quite work for you, you can modify the CloudFormation templates or build your own AMI. It doesn't try to be all things to all people, but it does try to accomodate by not being a black box.
Usage
Note: in this guide, all commands to be run from the shell are shown preceded by >
to indicate the shell prompt.
Keights has a git branch for each supported version of Kubernetes. For example, for Kubernetes 1.10.x
, the branch is v1.10
. Releases are numbered according to the version of Kubernetes with an incrementing suffix appended for each release supporting that Kubernetes version. For Kubernetes 1.10.0
, this would be v1.10.0-1
or v1.10.0-2
, for example.
The master
branch follows the latest release branch, so if you are deploying the latest release, proceed. Otherwise, do a git checkout of the release branch or desired release tag.
> git checkout v1.9
The stack/cloudformation
directory contains three CloudFormation templates: common.yml
, master.yml
, and node.yml
. The stacks must be deployed in that order.
common.yml
This contains security groups and IAM roles for the cluster.
Some of the outputs of the stack will be needed to pass to master.yml
and node.yml
.
These are: LoadBalancerSecurityGroup
, MasterSecurityGroup
, NodeSecurityGroup
, MasterInstanceProfile
, NodeInstanceProfile
, and LambdaRoleArn
.
master.yml
This contains the master stack. It will require as inputs the LoadBalancerSecurityGroup
, MasterSecurityGroup
, MasterInstanceProfile
, and LambdaRoleArn
outputs from the common.yml
stack.
The MasterSecurityGroup
output from common.yml
should go into the MasterSecurityGroups
parameter, though you may add additional groups to the list.
node.yml
This contains the node stack. You can deploy as many of these as you like, giving each one its own parameters for the node labels. It will require as inputs the NodeSecurityGroup
and NodeInstanceProfile
outputs from the common.yml
stack, and the LoadBalancerDnsName
output from the master.yml
stack.
The NodeSecurityGroup
output from common.yml
should go into the NodeSecurityGroups
parameter, though you may add additional groups to the list.
Deploy with Ansible
The stack/ansible
directory contains an Ansible role keights-stack
which deploys all CloudFormation stacks together, passing the outputs of one stack as inputs to another as needed. The role is published to GitHub on each keights
release.
A deploy
script is included in stack/ansible
which creates a Python virtualenv, installs Ansible into it, uses ansible-galaxy
to download the Ansible role, and then runs ansible-playbook
. You need either Python 3, or Python 2 with the virtualenv
command, installed and on your PATH
. Python 3 now includes virtual environments using the venv
standard library module.
The directory stack/ansible/example
contains the recommended layout, structured according to the expectations of the deploy
script.
To build a cluster, create a directory with the same layout as the example
directory. All of the identifiers in example/vars.yml
are fake, so edit the file to include a real VPC ID, subnet IDs, and so on. In requirements.yml
, the version of the Ansible role should match the keights
version.
Choose a name for the cluster. For our purpose, it is legbegbe
.
You also need AWS credentials and region to be in scope. You can do this with environment variables or a credentials file.
The cluster name must be set as an environment variable called CLUSTER
.
Now run deploy
from the directory you created:
> export AWS_DEFAULT_REGION=us-east-1
> export AWS_PROFILE=keights
> export CLUSTER=legbegbe
> /path/to/deploy
The deploy
scripts passes any additional arguments on to ansible-playbook
, so for example you can increase the logging by passing -vvv
to deploy
.
In a few minutes, your cluster should be ready.
Keeping up to date
Keep your cluster up to date by modifying any of the files in your directory and rerunning deploy
. For example, you may update the Kubernetes version by modifying the version of the keights-stack
role in requirements.yml
when there is a new keights
release. keights
strives to be able to make this transition smoothly, however always run such updates on a test cluster before running on a live system.
It is highly recommended to check your cluster's Ansible directory into source control, and let a CI/CD tool run deploy
when the source changes. The Ansible role is idempotent, and there should be no changes unless the source changes. It is important to keep all versions pinned in requirements.txt
and requirements.yml
, to avoid unforeseen side effects.
Connecting to the cluster
In the stack
directory is a script called kubesetup
. It sets up ~/.kube/config
for your cluster. To use it, you need both kubectl and the AWS CLI to be installed and on your PATH
, and the AWS region and credentials must be in scope.
Run kubesetup
with the cluster name as its only argument:
> export AWS_DEFAULT_REGION=us-east-1
> export AWS_PROFILE=keights
> ./kubesetup legbegbe
Now you can use kubectl to connect to the cluster.
> kubectl get no
AMI
A Debian AMI is created for each release containing the installed keights package and all dependencies, including required docker images.
The software should in theory run on any systemd based Linux distribution, such as RHEL, though it is currently only tested on Debian.