baghdad

README

Baghdad

Scalable CI for micro-services.

Intro

What if you could build 10, 50, 100, Docker images simultaneously at the same speed of building one image?

What if all of those images are in one repo, because your project is micro-services oriented?

What if you have more than one repo, with many Dockerfiles inside each one?

Baghdad runs within your swarm cluster, builds, versions, and deploys your micro-services, along with itself.

Baghdad also leverages every node within your cluster to parallelize Docker image builds.

Highlights

• Build and deploy multiple repos simultaneously.
• Build and deploy multiple services within the same repository, also simultaneously.
• Specify multiple Dockerfiles within the same repository, multi-stage builds supported.
• Automatically tag your repo and create releases on github.
• Automatically extract files from built images and push them as artifacts to github releases.
• Monitor your builds from the CLI.
• Monitor your app logs from the CLI.
• Check Pull Requests' build status with a simple UI.

Status

• Baghdad is not yet battle tested, and APIs might change until 1.0

Usage (simple, no config.)

1. Make sure you have a working Baghdad ecosystem, and that your Github Webhook is properly setup (see Deploy To Production).

2. Create a Dockerfile in your root folder.

3. Enjoy : )

4. All your commits to master will trigger an image build, and push to your image repository (Dockerhub or others), and will also create a pre-release on github. Pull Requests will trigger a build but without any tags/pushes to repos.

5. To deploy any of these releases run: bag deploy --env <env> --branch master --tag <tag>

Usage (detailed, with config)

1. For builds, you will need a baghdad.toml in your root directory. Refer to the full example for documentation.

2. For deploys, you will need a stack-compose.yml in your root directory. This is the docker-compose file that will be included in the docker stack deploy --compose-file... command. Baghdad runs on Baghdad, so you can see its own stack-compose.yml for reference. There are two things to note about this file:

   • You do not need to include a tag in the image property, as Baghdad will do that for you based on the deploy tag. For example say you have a stack-compose file with the following config:

     version: "3.2"
     services:
       web:
         image: my-org/my-web-image
     

     if you do bag deploy --tag master-1.0.0-Build.33, Baghdad will deploy your projec with the following stack file:

     version: "3.2"
     services:
       web:
         image: my-org/my-web-image:master-1.0.0-Build.33
     

   • Baghdad runs your stack within Traefik, so you do not need to specify those properties in your stack-compose.yml, Baghdad will do it for you.

Deploy to Production

1. Make sure you have at least one EC2 instance, with docker installed (v17.06.0-ce-rc2) and Swarm Mode enabled: docker swarm init

2. Create a traefik.toml file with the following config:

   [entryPoints]
   [entryPoints.http]
   address = ":80"
   [entryPoints.http.redirect]
   entryPoint = "https"
   [entryPoints.https]
   address = ":443"
   [entryPoints.https.tls]
   [[entryPoints.https.tls.certificates]]
   CertFile = "/ssl/cert.pem"
   KeyFile = "/ssl/key.pem”
   

   this assumes you have key/cert for https. If you don't, feel free to tweak the config above to only use https.

3. Deploy Traefik to the swarm

   docker service create \
       --name traefik \
       --detach=false \
       --constraint=node.role==manager \
       --publish 80:80 --publish 8080:8080 --publish 443:443 \
       --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
       --mount type=bind,source=$PWD/traefik.toml,target=/etc/traefik/traefik.toml \
       --mount type=bind,source=$PWD/cert.pem,target=/ssl/cert.pem \
       --mount type=bind,source=$PWD/key.pem,target=/ssl/key.pem \
       --network traefik-net \
       traefik:v1.3.0-rc3 \
       --docker \
       --docker.swarmmode \
       --docker.domain=traefik \
       --docker.watch \
       --logLevel=DEBUG \
       --web
   

4. Navigate to your Route53 or DNS provider and route *.YOUR_DOMAIN.COM to the IP address where Traefik is deployed.

5. From the Baghdad repo, run bag generate stack --env prod --host YOUR_DOMAIN.COM --version <pick-ur-release-tag>

6. Copy the stdout into the EC2 working direcory under stack.yml

7. Make sure to add the following secret file to the swarm under the secret name baghdad-vars:

ADMIN_TOKEN=<your-github-token> # will give Baghdad access to git clone your repo in order to build it.
DOCKER_ORG=<DOCKER_ORG_NAME> # where built docker images get puhsed.
DOCKER_AUTH_USER=<DOCKER_AUTH_USER>
DOCKER_AUTH_PASS=<DOCKER_AUTH_PASS>
BAGHDAD_DOMAIN_NAME=<DOMAIN_NAME> # example.com NOT www.example.com.

1. run docker stack deploy --compose-file stack.yml baghdad_prod

2. Create a github hook that points to https://master-prod-baghdad-api.YOUR_DOMAIN.COM/hooks/github. The passcode is baghdad (required). Make sure to disable SSL, if it's a self signed certificate.

3. Say a prayer ☪☮ℰ✡☥☯✝

Development

Baghdad consists of many services. You can instantiate all of them or any of them. The easiest way to do that, is through the docker-compose.yml file (not to be confused with stack-compose.yml). Each service defined in that file is geared for development mode. You only need to make sure you have a .env file in the root direcory with the same settings as baghdad-vars in the Deploy to Production section. Make sure to have BAGHDAD_DOMAIN_NAME=localhost

Make sure your your local docker daemon is in swarm mode.

Also, make sure you deploy traeffik as a swarm service with a different port than 80, and that's where you can test deployed projects:

docker service create \
    --name traefik \
    --detach=false \
    --constraint=node.role==manager \
    --publish 3456:80 --publish 9090:8080 \
    --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
    --network traefik-net \
    traefik:v1.3.0-rc3 \
    --docker \
    --docker.swarmmode \
    --docker.domain=traefik \
    --docker.watch \
    --logLevel=DEBUG \
    --web

Note that that I'm binding 3456 to the main router, and 9090 for the traefik web UI, because the Baghdad rabbitmq binds 8080.

Roadmap

• E2E tests.
• Automated memory monitoring & recovery.
• Document architecture.

Documentation

Index

• Constants
• func EncodeBuildEvent(be BuildEvent) (p []byte, err error)
• func EncodeBuildJob(bj BuildJob) (p []byte, err error)
• func EncodeDeployJob(dj DeployJob) (p []byte, err error)
• func EncodePostDeployJob(pdj PostDeployJob) (p []byte, err error)
• func EncodeSecretsJob(sj SecretsJob) (p []byte, err error)
• type Baghdad
• type Branch
• type BuildEvent
  • func DecodeBuildEvent(p []byte) (be BuildEvent, err error)
• type BuildJob
  • func DecodeBuildJob(p []byte) (bj BuildJob, err error)
• type DeployJob
  • func DecodeDeployJob(p []byte) (dj DeployJob, err error)
• type PostDeploy
• type PostDeployJob
  • func DecodePostDeployJob(p []byte) (pdj PostDeployJob, err error)
• type SecretsJob
  • func DecodeSecretsJob(p []byte) (sj SecretsJob, err error)
• type Service

Constants

const (
	// PushEvent enum
	PushEvent = 1 + iota
	// PullRequestEvent enum
	PullRequestEvent
	// BuildSuccessEvent enum
	BuildSuccessEvent
	// BuildFailureEvent enum
	BuildFailureEvent
)

Functions

func EncodeBuildEvent

func EncodeBuildEvent(be BuildEvent) (p []byte, err error)

EncodeBuildEvent is used to serialize a BuildEvent to a rabbitmq msg

func EncodeBuildJob

func EncodeBuildJob(bj BuildJob) (p []byte, err error)

EncodeBuildJob is used to serialize a BuildJob for a rabbitmq msg

func EncodeDeployJob

func EncodeDeployJob(dj DeployJob) (p []byte, err error)

EncodeDeployJob is used to serialize a DeployJob to a rabbitmq msg

func EncodePostDeployJob

func EncodePostDeployJob(pdj PostDeployJob) (p []byte, err error)

EncodePostDeployJob is used to serialize a PostDeployJob to a rabbitmq msg

func EncodeSecretsJob

func EncodeSecretsJob(sj SecretsJob) (p []byte, err error)

EncodeSecretsJob is used to serialize a SecretsJob to a rabbitmq msg

Types

type Baghdad

type Baghdad struct {
	Project      string            `toml:"project"`
	SlackURL     string            `toml:"slack-url"`
	Services     []Service         `toml:"services"`
	Environments map[string]string `toml:"environments"`
	Branches     map[string]Branch `toml:"branches"`
	PostDeploy   PostDeploy        `toml:"post-deploy"`
}

Baghdad top level config

type Branch

type Branch struct {
	Version string `toml:"version"`
}

Branch config

type BuildEvent

type BuildEvent struct {
	BuildJob
	EventType int
	Tag       string
	LogID     string
}

BuildEvent is used at the end of a build to let subscribes know if a build failed/succeeded.

func DecodeBuildEvent

func DecodeBuildEvent(p []byte) (be BuildEvent, err error)

DecodeBuildEvent is used to parse a BuildEvent from a rabbitmq msg

type BuildJob

type BuildJob struct {
	Baghdad    Baghdad
	BranchName string
	GitURL     string
	PRNum      int
	RepoName   string
	RepoOwner  string
	SHA        string
	Type       int // use PushEvent/PullEvent enums
	Service    Service
	NextTag    string
	ErrString  string
	LogID      string
}

BuildJob use to send a build job to the build worker

func DecodeBuildJob

func DecodeBuildJob(p []byte) (bj BuildJob, err error)

DecodeBuildJob is used to parse a BuildJob from a rabbitmq msg

type DeployJob

type DeployJob struct {
	Baghdad    Baghdad
	BranchName string
	Env        string
	RepoName   string
	Tag        string
	RepoOwner  string
	LogID      string
}

DeployJob use to send a deploy job to the deploy worker

func DecodeDeployJob

func DecodeDeployJob(p []byte) (dj DeployJob, err error)

DecodeDeployJob is used to parse a DeployJob from a rabbitmq msg

type PostDeploy

type PostDeploy struct {
	SourceService string   `toml:"source-service"`
	TargetService string   `toml:"target-service"`
	Secrets       string   `toml:"secrets"`
	Environments  []string `toml:"environments"`
}

PostDeploy config

type PostDeployJob

type PostDeployJob struct {
	Baghdad     Baghdad
	ProjectName string
	Tag         string
	Env         string
	BranchName  string
	SiteURL     string
}

PostDeployJob use to send a post deploy job to rabbitmq

func DecodePostDeployJob

func DecodePostDeployJob(p []byte) (pdj PostDeployJob, err error)

DecodePostDeployJob is used to parse a DeployJob from a rabbitmq msg

type SecretsJob

type SecretsJob struct {
	ProjectName string
	SecretName  string
	SecretBody  []byte
}

SecretsJob use to save a secret to swarm.

func DecodeSecretsJob

func DecodeSecretsJob(p []byte) (sj SecretsJob, err error)

DecodeSecretsJob is used to parse a SecretsJob from a rabbitmq msg

type Service

type Service struct {
	Name          string `toml:"name"`
	Dockerfile    string `toml:"dockerfile"`
	IsExposed     bool   `toml:"isExposed"`
	Port          string `toml:"port"`
	Image         string `toml:"image"`
	IsExternal    bool   `toml:"isExternal"`
	HasArtifacts  bool   `toml:"hasArtifacts"`
	ArtifactsPath string `toml:"artifactsPath"`
}

Service config