decap

README

History

Mark: How hard can it be to build a build server?

Josh: How hard can space travel be? But no matter. So why not use a pure Kubernetes backend?

Overview

Decap is a CI build server based on a Kubernetes backend that executes shell script-based builds in a specially provided build pod. The backend is a containerized webapp that manages build pods, and the frontend a single page app that provides a friendly UX. Builds are executed in pods spun up on demand on a per-build basis, with build results published to S3 buckets and a DynamoDb table.

Decap was originally conceived as a headless build server, driven solely by post commit hooks from userland build repositories. It now has a frontend, but the name stuck.

This project is under active development, and has no releases yet.

Theory of Operation

You have projects you want to build. Your builds are articulated in terms of userland shell scripts. Decap ships with a base build container that mounts your build scripts as a git repository and locates them by a team/libary convention in the container filesystem.

Either user-initiated builds or post commit hooks sent from your projects of interest drive HTTP requests to the containerized Decap webapp. This webapp in turn makes calls to the Kubernetes API master to launch an ephemeral build pod to build a single instance of your code. Once the build is finished the pod exits, saving no build pod state from one build to the next. Build results are shipped to Amazon AWS S3 buckets and a DynamoDb table.

Your build scripts are completely free form. Here are two examples:

#!/bin/bash

echo hello world

#!/bin/bash

git clone https://git.example.com/repo --branch ${BRANCH_TO_BUILD}
mvn clean install
mvn deploy

Sidecar build containers

If your build needs additional services, such as MySQL, RabbitMQ, etc., Decap provides a way to ingest a set of additional container specs into the Kubernetes build Pod descriptor. This means those services will be available to your build at localhost:port.

AWS Setup

Decap uses AWS S3 buckets to store build artifacts and console logs, and DynamoDb to store overall build results and metadata.

To run the scripts below that create these AWS resources, you must have installed the AWS Command Line Client. For instructions on how to do this, see http://aws.amazon.com/documentation/cli/.

jq is also installation requirement, and is needed to parse certain outputs of the Amazon aws command line tool.

IAM user

In ./aws-resources we provide shell scripts for creating the AWS resources Decap needs. To run these scripts effectively, you will need an AWS account with what we're calling root like powers. This account must be capable of creating the following types of resources on AWS:

• AWS IAM users
• Access credentials
• S3 buckets
• DynamoDb tables
• Policies

Your main AWS Dashboard account should have these powers. If it does not, contact your AWS administrator.

N.B. Your Dashboard account is used only in a one-time capacity to create these resources. Thereafter, Decap is configured to use these resources, including the created IAM user decap. Your Dashboard account is not used by Decap at runtime.

One time admin configuration

Using a profile name decapadmin, add your AWS Dashboard account Access Key ID and Secret Access Key to $HOME/.aws/credentials file. Add a default region to $HOME/.aws/config.

$HOME/.aws/credentials:

[decapadmin]
aws_access_key_id = (your Dashboard key)
aws_secret_access_key = (your Dashboard secret)

$HOME/.aws/config:

[decapadmin]
region=us-west-1

Create the AWS resources Decap requires

$ cd aws-resources
$ sh create-world.sh

You should now be able to view the following resources in your AWS Dashboard UI:

• an IAM user named decap
• a set of access credentials for use by the newly created decap user, written to the file aws.credentials
• two S3 buckets: decap-console-logs and decap-build-artifacts
• one DynamoDb table named decap-build-metadata
• five policies attached to the user decap, which are required for access to the S3 buckets and DynamoDb table

The five policies are named:

• decap-db-base
• decap-db-isBuilding
• decap-db-projectKey
• decap-s3-build-artifacts
• decap-s3-console-logs

Kubernetes Cluster Setup

Bring up a Kubernetes cluster as appropriate: https://github.com/kubernetes/kubernetes/tree/master/docs/getting-started-guides. Decap requires SkyDNS, which is included if you use the standard kube-up.sh script to create your cluster.

Namespaces

Decap requires two Kubernetes namespaces: decap and decap-system. The decap namespace is where your build pods run. decap-system is where the containerized Decap webapp runs.

Create the Kubernetes namespaces required for Decap

$ kubectl create -f k8s-resources/decap-namespaces.yaml

Decap Kubernetes Secret for AWS and Github credentials

The AWS Access Key and Secret for user decap created above allows the build pod to upload build artifacts and console logs to S3, and to write build information to the DynamoDb table. The Access Key and Secret also allows the Decap webapp to access these same buckets and table.

To be most effective, Decap also needs access to the list of branches for your various projects. Decap can query your project repositories for this branch information. Without access to your project branch information, Decap's web UI cannot offer to build a particular branch on your projects. For Github projects, this means Decap needs an OAuth2 Github ClientID and ClientSecret. Generate Github OAuth2 credentials here: https://github.com/settings/applications/new.

Using the AWS Access Key and Secret in ./aws-resources/aws.credentials, and your Github ClientID and ClientSecret, craft a k8s-resources/decap-secrets.yaml

apiVersion: v1
data:
  aws-key: thekey
  aws-secret: base64(thesecret)
  aws-region: base64(theregion)
  github-client-id: base64(github client-id)
  github-client-secret: base64(github client-secret)
kind: Secret
metadata:
     name: decap-credentials
type: Opaque

and create it on the Kubernetes cluster in both the decap and decap-system namespaces

$ kubectl --namespace=decap-system create -f k8s-resources/decap-secrets.yaml
$ kubectl --namespace=decap create -f k8s-resources/decap-secrets.yaml

The base build container will automatically have these Kubernetes Secrets mounted in both the build container and the webapp container. The base build container will use them for publishing build results. The webapp container will use them for querying AWS and Github.

Decap Kubernetes Pod creation

Create the pod that runs the Decap webapp in the cluster:

$ kubectl create -f k8s-resources/decap.yaml

Setting up a build scripts repository

Decap leverages Kubernetes's ability to mount a Git repository readonly inside a container. When you launch a build in the build container, Kubernetes will mount the build scripts repo that contains the build scripts for your projects. Here is a sample build script repository

https://github.com/ae6rt/decap-build-scripts

The build container refers to this repository as a mounted volume. Build scripts are indexed by project key by the build container entrypoint. For github based projects, the project key is the github username + "/" + repository name. Generally, the username is referred to as the team and the repository basename as the project For example, if the github username is ae6rt and the repository name is dynamodb-lab, then the project key is "ae6rt/dynamodb-lab". The build script is by convention named build.sh and is located relative to the top level of the build scipts repository at ae6rt/dynamodb-lab/build.sh.

The build container will call your project's build script, capture the console logs, and ship the build artifacts, console logs and build metadata to S3 and DynamoDb.

Project metadata files and branch information

Each project must have a project.json file placed on par with a project's build.sh script. Projects omitting the file will be ignored by Decap. project.json has the following format

{
     "buildImage": "ae6rt/decap-build-base:latest",
     "repoManager": "github",
     "managedRefRegex": "master|develop|issue/.*",
     "repoUrl": "https://github.com/ae6rt/dynamodb-lab.git",
     "repoDescription": "AWS DynamoDb lab"
}

All project.json files must have a buildImage field. Without it, decap cannot pull the container image in which to execute the build.

Knowing the repository manager and repository URL, Decap can query the repository manager for branches and tags on the project. Knowing the branches, the Decap web UI can offer to let the user build a particular branch on a project. Github is currently the only supported repository manager.

The project json descriptor contains an optional field managedRefRegex. If this field is present, only branches and tags that match the regex will be built as a result of post-commit hook handling of a project git-push. If this field is omitted, all refs are eligible to be built in the event of a post-commit push on the project. The managedRefRegex does not affect which branches can be built manually through the Decap web UI.

Sidecar build containers

TBD

Handling updates to the build scripts repository

Decap will refresh its representation of the build scripts repository if you add a post-commit hook to the build scripts repository. Point the post commit URL at Decap baseURL/hooks/buildscripts. Any HTTP POST to this endpoint will force a refresh of the build script repository in the Decap webapp.

Base Build Container Environment

Here is the base build container reference: https://github.com/ae6rt/decap/tree/master/build-container

The following environment variables are available in your build scripts:

• BUILD_ID: UUID that uniquely identifies this build
• PROJECT_KEY: a composite key consisting of your project team/project
• BRANCH_TO_BUILD: an optional git branch for use with builds that can put it to use

Concurrent builds of a given team/project + branch are currently forbidden, and enforced with a lock in etcd, which runs in the same pod as the Decap webapp.

Build pod instances are given the following Kubernetes labels

"labels": {
   "type": "decap-build",
   "team": "{{.Team}}",
   "project": "{{.Project}}",
   "branch": "{{.BranchToBuild}}",
}

Developing Decap

The Decap source is divided into three parts:

• Base Build Container in build-container/
• Webapp in web/
• Kubernetes resource configs in k8s-resources/

Base Build Container

This is the place to modify the base build container ENTRYPOINT script and Dockerfile

Webapp

This is a Go webapp that receives commit hooks from various repository managers. Upon receiving a hook on a managed project, Decap will launch a container to execute a build on the project and branch.

Kubernetes resource configs

This contains yaml files that describe Kubernetes resources Decap needs to function.