README ¶
cicd
cicd is a simple command line tool that facilitates build and deployment for your project. The goal is to help enable developers to easily setup a continuous build pipeline using GitLab CI/CD and code driven deployment.
Overview
The command line tool provides the functionality to configure, build and deploy your code. When new code is push to GitLab, this tool will enable building, testing and deploying your code to Amazon AWS.
Deploying your code to production always requires additional tooling and configuration. Instead of patching together a system of of existing tools and configuration files. This tool centralizes configuration for the application and any additional deployment resources needed.
Configuration is define with code. AWS resources are created/maintained using the AWS SDK for Go.
This tool is used by GitLab CI/CD and is configured by a file called .gitlab-ci.yml placed at the repository’s root.
All code is deployed to Amazon AWS.
Check out the full presentation that covers how to setup your GitLab CI/CD pipeline that uses autoscaling GitLab Runners on AWS.
Support is provided for both services and functions. The build process for both relies on docker and thus, neither are required to be written in go.
Configuration for build and deploy is provided by gitlab.com/geeks-accelerator/oss/devops/pkg/devdeploy
For additional details regarding this tool, refer to gitlab.com/geeks-accelerator/oss/devops
Deployment Environments
All configuration for the deployment environments is defined in code that is located in the internal/config package. Multiple development environments can easily be configured for more control.
This tool supports three target deployment environments:
- dev
- stage
- prod
Additional deployment resources will need be configured:
- S3 buckets
- RDS postgres database
- Redis elastic cache cluster
Secrets and other credentials are stored in AWS Secrets Manager.
Services
Services are generally applications that will need to be long running or continuously available. An example web API deployed as service is provided by the devops project in [examples]((https://gitlab.com/geeks-accelerator/oss/devops/tree/master/examples).
The Dockerfile for
the example service is defined as multi-stage build
that includes building a base layer, running unittests and compiling the go application as static binary. The final
layer in the multi-stage uses alpine:3.9 as its base image and copies
in the compiled binary resulting in a docker container that is around 50mbs excluding any additional static assets. It's
possible to swap out alpine:3.9
with busybox
for an even small resulting docker image.
A service is built using the defined Dockerfile. The resulting image is pushed to Amazon Elastic Container Registry.
Amazon Elastic Container Registry (ECR) is a fully-managed Docker container registry that makes it easy for
developers to store, manage, and deploy Docker container images. Amazon ECR is integrated with Amazon Elastic
Container Service (ECS) simplifying the development to production workflow.
A service is configured for deployment in services.go. Services are deployed to AWS Fargate based on the defined task definition.
AWS Fargate is a compute engine for Amazon ECS that allows you to run containers without having to manage servers or
clusters. With AWS Fargate, you no longer have to provision, configure, and scale clusters of virtual machines to
run containers.
If the docker file is a multi-stage build and it contains a stage with the name build_base_golang
, additional caching will
be implemented to reduce build times. The build command assumes for a stage named build_base_golang
assumes that the
stage will run go mod download
to pull down all package dependencies. The build command computes a checksum for the
project go.mod and then executes a docker build that targets the specific stage build_base_golang
. The built container
image is tagged with the go.mod hash and pushed to the projects
GitLab repository.
Functions
Functions are applications that can be executed in short period of time. An python script for Datadog Log Collection deployed as a function is provided by the devops project in [examples]((https://gitlab.com/geeks-accelerator/oss/devops/tree/master/examples).
A function is built using the defined Dockerfile.
The Dockerfile
should use a lambdaci image as the base image.
Lambdaci images provide a sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries, file structure and permissions, environment variables, context objects and behaviors – even the user and running process are the same.
The build command then uses docker cp
to extract all files from the resulting container image that are located in
/var/task
. These files are zipped and uploaded to AWS S3 for deployment.
A function is configured for deployment in functions.go. Functions are deployed to AWS Lambda.
AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume
- there is no charge when your code is not running.
Schema Migrations
cicd includes a minimalistic database migration script that implements github.com/geeks-accelerator/sqlxmigrate. It provides schema versioning and migration rollback. The schema for the entire project is defined globally and is located inside internal: internal/schema
The example schema package provides two separate methods for handling schema migration:
-
Migrations - List of direct SQL statements for each migration with defined version ID. A database table is created to persist executed migrations. Upon run of each schema migration run, the migration logic checks the migration database table to check if it’s already been executed. Thus, schema migrations are only ever executed once. Migrations are defined as a function to enable complex migrations so results from query manipulated before being piped to the next query.
-
Init Schema - If you have a lot of migrations, it can be a pain to run all them. For example, when you are deploying a new instance of the app into a clean database. To prevent this, use the initSchema function that will run as-if no migration was run before (in a new clean database).
Another bonus with the globally defined schema is that it enables your testing package the ability to dynamically spin up database containers on-demand and automatically include all the migrations. This allows the testing package to programmatically execute schema migrations before running any unit tests.
Installation
Make sure you have a working Go environment. Go version 1.2+ is supported. See the install instructions for Go.
To install cicd, simply run:
$ go get gitlab.com/geeks-accelerator/oss/devops/build/cicd
Make sure your PATH
includes the $GOPATH/bin
directory so your commands can
be easily used:
export PATH=$PATH:$GOPATH/bin
Getting Started
cicd requires AWS permissions to be executed locally. For the GitLab CI/CD build pipeline, AWS roles will be used. This user is only nessissary for running cicd locally.
-
You will need an existing AWS account or create a new AWS account.
-
Define a new AWS IAM Policy called
saas-starter-kit-deploy
with a defined JSON statement instead of using the visual editor. The statement is rather large as each permission is granted individually. A copy of the statement is stored in the devops repo at configs/aws-aim-deploy-policy.json -
Create new AWS User called
saas-starter-kit-deploy
with Programmatic Access and Attach existing policies directly with the policy created from step 1saas-starter-kit-deploy
-
Try running the build for a single service.
cicd --env=dev build service --name=aws-ecs-go-web-api --release-tag=testv1
- Try running the deploy for a single service.
cicd --env=dev deploy service --name=aws-ecs-go-web-api --release-tag=testv1
Usage
$ cicd [global options] command [command options] [arguments...]
Global Options
-
Target Environment - required
--env [dev|stage|prod]
-
AWS Access Key - optional or can be set via env variable
AWS_ACCESS_KEY_ID
--aws-access-key value
-
AWS Secret Key - optional, can be set via env variable
AWS_SECRET_ACCESS_KEY
--aws-secret-key value
-
AWS Region - optional, can be set via env variable
AWS_REGION
--aws-region value
-
AWS Use Role - optional, can be set via env variable
AWS_USE_ROLE
, when enabled an IAM Role else AWS Access/Secret Keys are required -
Show help
--help, -h
-
Print the version
--version, -v
Commands
-
build service
- Executes a build for a single service$ cicd -env [dev|stage|prod] build service -name NNNNN [command options]
Options:
--name value, -n value target service, required --release-tag value, --tag value optional tag to override default CI_COMMIT_SHORT_SHA --dry-run print out the build details --no-cache skip caching for the docker build --no-push disable pushing release image to remote repository
-
build function
- Executes a build for a single function$ cicd -env [dev|stage|prod] build function -name NNNNN [command options]
Options:
--name value, -n value target function, required --release-tag value, --tag value optional tag to override default CI_COMMIT_SHORT_SHA --dry-run print out the build details --no-cache skip caching for the docker build --no-push disable pushing release image to remote repository
-
deploy service
- Executes a deploy for a single service$ cicd -env [dev|stage|prod] deploy service -name NNNNN [command options]
Options:
--name value, -n value target service, one of [aws-ecs-go-web-api] --release-tag value, --tag value optional tag to override default CI_COMMIT_SHORT_SHA --dry-run print out the deploy details
-
deploy function
- Executes a deploy for a single function$ cicd -env [dev|stage|prod] deploy function -name NNNNN [command options]
Options:
--name value, -n value target function, required --release-tag value, --tag value optional tag to override default CI_COMMIT_SHORT_SHA --dry-run print out the deploy details
-
schema
- Runs the database migration using credentials from AWS Secrets Manager.$ cicd -env [dev|stage|prod] schema
-
help
- Shows a list of commands$ cicd help
Or for one command:
$ cicd build help
Examples
Build the example service aws-ecs-go-web-api
$ cicid --env=dev build service --name=aws-ecs-go-web-api --release-tag=testv1 --dry-run=false
Deploy the example service aws-ecs-go-web-api
$ cicid --env=dev deploy service --name=aws-ecs-go-web-api --release-tag=testv1 --dry-run=false
Join us on Gopher Slack
If you are having problems installing, troubles getting the project running or would like to contribute, join the channel #saas-starter-kit on Gopher Slack
Documentation ¶
There is no documentation for this package.