cali

README

Cali - Cali Automation Layout Initialiser

Oh, you found this repo, huh?

Well, fair warning, we're still figuring out how the open source version of Cali should work.

Feel free to play with it, but while we're still on v0.x.x, be sure to pin to specific versions in your dependency manager of choice.

---

Overview

Ever wanted to be able to ensure that all your developers are working using the same versions of the same tools?

Want to be free from dowloading huge Vagrant boxes?

Want to be able to update and distribute new versions of your tooling quickly and easily?

The answer to all these questions at our company was a resounding YES!

We needed a way to reduce the overheads associated with onboarding new starters, so instead of a document giving a list of software that needed to be downloaded, e.g. ChefDK, Vagrant, Terraform etc. etc. they could just download a single thing which gave them a ready made development environment whether they used Mac or Linux PCs. Vagrant was the obvious choice and for a while it worked, but as we scaled up as a tech team both in numbers and in diversity of tools, this quickly became unmanageable. Different teams started wanting different boxes with different thing and different versions of those things. We soon ended up with several huge box files to maintain, they were slow to update, slow to upload and slow to download and horrible to maintain. We spent a lot of time troubleshooting...

"Oh yes I have seen that error before, are you using Vagrant version X with VirtualBox version Y?

"Err... not sure this will work with Windows sorry!"

Then along came Docker for Mac and shortly after, Docker for Windows and this opened the world of containers up to non-Linux users and gave us the ability to distribute our tools using Docker across Mac, Linux and Windows for the first time. Being a Chef house, the first experiment was to run ChefDK out of a docker container and was distributed as some bash aliases (early days!).

alias buildtools="docker run --rm -it -v \$PWD:/root/build -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.aws:/root/.aws -e \"BUILD_ROOT=\$PWD\" -v ~/.gitconfig:/root/.gitconfig ourprivatedockerregistry.io/buildtools:stable"
alias kitchen="buildtools kitchen"

Typing kitchen would then execute Test Kitchen for Chef against your $PWD and no matter what laptop you were using, as long as you had docker, all our devs were now using the same version of ChefDK for developing Chef code against.

Clearly this was suboptimal and as soon as more tools came along such as Terraform which wanted to be able to obtain short leased AWS credentials from HashiCorp Vault, a new solution was needed to distribute and orchestrate these containers was needed. Also, would it not be cool if that same tool could be run directly, or as a container on CI? Then your developers and your CI is using exactly the same stuff?

So we broke out those Go ninja skills and started using the Docker API to programatically manage these containers, remembering at each step that at some point in the future, this tool would itself be containerised and and would need to be able to schedule its ephemeral job containers either directly on the host its running on, or tantalisingly, on a swarm. Each tool, whether it be ChefDK or Terraform or whatever would either need to be able to work with $PWD or be able to check out some git repo and do its thing within that clone, all within containers. The final pieces of the puzzle were including auth with Hashicorp Vault to allow us to connect our corporate AD to get on-demand AWS credentials and also an update command which would self update the code when a new version was available on our internal Artifact repository.

This has been a great success and we are really happy with how it works and how little day to day support we have to provide our developers to use it. Its been such a success that we are getting requests to put functionality into it which we feel does not really belong there. Rather than saying NO! to such feature requests, we are doing the only sane thing and open sourcing the guts of the system to allow anybody to go and create their own development tools to do whatever they want.

Building a CLI tool

package main

import "github.com/skybet/cali"

func main() {
	cli := cali.NewCli("cali")
	cli.SetShort("Example CLI tool")
	cli.SetLong("A nice long description of what your tool actually does")

	cmdTerraform(cli)

	cli.Start()
}

func cmdTerraform(cli *cali.Cli) {

	terraform := cli.NewCommand("terraform [command]")
	terraform.SetShort("Run Terraform in an ephemeral container")
	terraform.SetLong(`Starts a container for Terraform and attempts to run it against your code. There are two choices for code source; a local mount, or directly from a git repo.

Examples:

  To build the contents of the current working directory using my_account as the AWS profile from the shared credentials file on this host.
  # cali terraform plan -p my_account

  Any addtional flags sent to the terraform command come after the --, e.g.
  # cali terraform plan -- -state=environments/test/terraform.tfstate -var-file=environments/test/terraform.tfvars
  # cali terraform -- plan -out plan.out
`)
	terraform.Flags().StringP("profile", "p", "default", "Profile to use from the AWS shared credentials file")
	terraform.BindFlags()

	terraformTask := terraform.Task("hashicorp/terraform:0.9.9")
	terraformTask.SetInitFunc(func(t *cali.Task, args []string) {
		t.AddEnv("AWS_PROFILE", cli.FlagValues().GetString("profile"))
	})
}

Now you can run containerised terraform :D

$ example terraform init
Terraform initialized in an empty directory!

The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.

Or build from a git repo...

$ example terraform plan --git git@github.com:someone/terraform_code.git --git-branch master --git-path path/to/code

Docs

The API docs are available on GoDoc or take a look (and help edit) the wiki.

Documentation

Index

• type Cli
  • func NewCli(n string) *Cli
  • func (c *Cli) FlagValues() *viper.Viper
  • func (c *Cli) NewCommand(n string) *Command
  • func (c *Cli) Start()
• type Command
  • func (c *Command) BindFlags()
  • func (c *Command) Flags() *flag.FlagSet
  • func (c *Command) SetLong(l string)
  • func (c *Command) SetShort(s string)
  • func (c *Command) Task(def interface{}) *Task
• type CreateResponse
• type DockerClient
  • func NewDockerClient() *DockerClient
  • func (c *DockerClient) AddBind(bnd string)
  • func (c *DockerClient) AddBinds(bnds []string)
  • func (c *DockerClient) AddEnv(key, value string)
  • func (c *DockerClient) AddEnvs(envs []string)
  • func (c *DockerClient) BindFromGit(cfg *GitCheckoutConfig, noGit func() error) error
  • func (c *DockerClient) ContainerExists(name string) (bool, error)
  • func (c *DockerClient) DeleteContainer(id string) error
  • func (c *DockerClient) Git() *Git
  • func (c *DockerClient) ImageExists(image string) (bool, error)
  • func (c *DockerClient) InitDocker() error
  • func (c *DockerClient) Privileged(p bool)
  • func (c *DockerClient) PullImage(image string) error
  • func (c *DockerClient) SetBinds(bnds []string)
  • func (c *DockerClient) SetCmd(cmd []string)
  • func (c *DockerClient) SetConf(co *container.Config)
  • func (c *DockerClient) SetDefaults()
  • func (c *DockerClient) SetEnvs(envs []string)
  • func (c *DockerClient) SetHostConf(h *container.HostConfig)
  • func (c *DockerClient) SetImage(img string)
  • func (c *DockerClient) SetNetConf(n *network.NetworkingConfig)
  • func (c *DockerClient) SetWorkDir(wd string)
  • func (c *DockerClient) StartContainer(rm bool, name string) (string, error)
• type Event
• type Git
  • func (g *Git) Checkout(cfg *GitCheckoutConfig) (string, error)
  • func (g *Git) Pull(name string) (string, error)
• type GitCheckoutConfig
  • func (cfg GitCheckoutConfig) GetContainerName() (string, error)
• type ProgressDetail
• type Task
  • func (t *Task) Bind(src, dst string) (string, error)
  • func (t *Task) SetDefaults(args []string) error
  • func (t *Task) SetFunc(f TaskFunc)
  • func (t *Task) SetInitFunc(f TaskFunc)
• type TaskFunc

Types

type Cli

type Cli struct {
	Name string

	*Command
	// contains filtered or unexported fields
}

Cli is the application itself

func NewCli

func NewCli(n string) *Cli

NewCli returns a brand new cli

func (*Cli) FlagValues

func (c *Cli) FlagValues() *viper.Viper

FlagValues returns the wrapped viper object allowing the API consumer to use methods like GetString to get values from config

func (*Cli) NewCommand

func (c *Cli) NewCommand(n string) *Command

NewCommand returns a brand new command attached to it's parent cli

func (*Cli) Start

func (c *Cli) Start()

Start the fans please!

type Command

type Command struct {
	RunTask *Task
	// contains filtered or unexported fields
}

Command is the actual command run by the cli and essentially just wraps cobra.Command and has an associated Task

func (*Command) BindFlags

func (c *Command) BindFlags()

BindFlags needs to be called after all flags for a command have been defined

func (*Command) Flags

func (c *Command) Flags() *flag.FlagSet

Flags returns the FlagSet for the command and is used to set new flags for the command

func (*Command) SetLong

func (c *Command) SetLong(l string)

SetLong sets the long description of the command

func (*Command) SetShort

func (c *Command) SetShort(s string)

SetShort sets the short description of the command

func (*Command) Task

func (c *Command) Task(def interface{}) *Task

Task is something executed by a command

type CreateResponse

type CreateResponse struct {
	ID             string         `json:"id"`
	Status         string         `json:"status"`
	ProgressDetail ProgressDetail `json:"progressDetail"`
	Progress       string         `json:"progress,omitempty"`
}

CreateResponse is the response from Docker API when pulling an image

type DockerClient

type DockerClient struct {
	Cli      *client.Client
	HostConf *container.HostConfig
	NetConf  *network.NetworkingConfig
	Conf     *container.Config
}

DockerClient is a slimmed down implementation of the docker cli

func NewDockerClient

func NewDockerClient() *DockerClient

NewDockerClient returns a new DockerClient initialised with the API object

func (*DockerClient) AddBind

func (c *DockerClient) AddBind(bnd string)

AddBind adds a bind mount to the HostConfig

func (*DockerClient) AddBinds

func (c *DockerClient) AddBinds(bnds []string)

AddBinds adds multiple bind mounts to the HostConfig

func (*DockerClient) AddEnv

func (c *DockerClient) AddEnv(key, value string)

AddEnv adds an environment variable to the HostConfig

func (*DockerClient) AddEnvs

func (c *DockerClient) AddEnvs(envs []string)

AddEnvs adds multiple envs to the HostConfig

func (*DockerClient) BindFromGit

func (c *DockerClient) BindFromGit(cfg *GitCheckoutConfig, noGit func() error) error

BindFromGit creates a data container with a git clone inside and mounts its volumes inside your app container If there is no valid Git repo set in config, the noGit callback function will be executed instead

func (*DockerClient) ContainerExists

func (c *DockerClient) ContainerExists(name string) (bool, error)

ContainerExists determines if the container with this name exist

func (*DockerClient) DeleteContainer

func (c *DockerClient) DeleteContainer(id string) error

DeleteContainer - Delete a container

func (*DockerClient) Git

func (c *DockerClient) Git() *Git

Git returns a new instance

func (*DockerClient) ImageExists

func (c *DockerClient) ImageExists(image string) (bool, error)

ImageExists determines if an image exist locally

func (*DockerClient) InitDocker

func (c *DockerClient) InitDocker() error

InitDocker initialises the client

func (*DockerClient) Privileged

func (c *DockerClient) Privileged(p bool)

Privileged sets whether the container should run as privileged

func (*DockerClient) PullImage

func (c *DockerClient) PullImage(image string) error

PullImage - Pull an image locally

func (*DockerClient) SetBinds

func (c *DockerClient) SetBinds(bnds []string)

SetBinds sets the bind mounts in the HostConfig

func (*DockerClient) SetCmd

func (c *DockerClient) SetCmd(cmd []string)

SetCmd sets the command to run in the container

func (*DockerClient) SetConf

func (c *DockerClient) SetConf(co *container.Config)

SetConf sets the container.Config struct for the new container

func (*DockerClient) SetDefaults

func (c *DockerClient) SetDefaults()

SetDefaults sets container, host and net configs to defaults. Called when instantiating a new client or can be called manually at any time to reset API configs back to empty defaults

func (*DockerClient) SetEnvs

func (c *DockerClient) SetEnvs(envs []string)

SetEnvs sets the environment variables in the Conf

func (*DockerClient) SetHostConf

func (c *DockerClient) SetHostConf(h *container.HostConfig)

SetHostConf sets the container.HostConfig struct for the new container

func (*DockerClient) SetImage

func (c *DockerClient) SetImage(img string)

SetImage sets the image in Conf

func (*DockerClient) SetNetConf

func (c *DockerClient) SetNetConf(n *network.NetworkingConfig)

SetNetConf sets the network.NetworkingConfig struct for the new container

func (*DockerClient) SetWorkDir

func (c *DockerClient) SetWorkDir(wd string)

SetWorkDir sets the working directory of the container

func (*DockerClient) StartContainer

func (c *DockerClient) StartContainer(rm bool, name string) (string, error)

StartContainer will create and start a container with logs and optional cleanup

type Event

type Event struct {
	ID     string `json:"id"`
	Status string `json:"status"`
}

Event holds the json structure for Docker API events

type Git

type Git struct {
	Image string
	// contains filtered or unexported fields
}

Git is used to interact with containerised git

func (*Git) Checkout

func (g *Git) Checkout(cfg *GitCheckoutConfig) (string, error)

Checkout will create and start a container, checkout repo and leave container stopped so volume can be imported

func (*Git) Pull

func (g *Git) Pull(name string) (string, error)

Pull will perform a git pull in a git repo container

type GitCheckoutConfig

type GitCheckoutConfig struct {
	Repo, Branch, RelPath, Image string
}

GitCheckoutConfig is input for Git.Checkout

func (GitCheckoutConfig) GetContainerName

func (cfg GitCheckoutConfig) GetContainerName() (string, error)

GetContainerName returns a container name for provided Git config

type ProgressDetail

type ProgressDetail struct {
	Current int `json:"current,omitempty"`
	Total   int `json:"total,omitempty"`
}

ProgressDetail records the progress achieved downloading an image

type Task

type Task struct {
	*DockerClient
	// contains filtered or unexported fields
}

Task is the action performed when it's parent command is run

func (*Task) Bind

func (t *Task) Bind(src, dst string) (string, error)

Bind is a utility function which will return the correctly formatted string when given a source and destination directory

The ~ symbol and relative paths will be correctly expanded depending on the host OS

func (*Task) SetDefaults

func (t *Task) SetDefaults(args []string) error

SetDefaults sets the default host config for a task container Mounts the PWD to /tmp/workspace Mounts your ~/.aws directory to /root - change this if your image runs as a non-root user Sets /tmp/workspace as the workdir Configures git

func (*Task) SetFunc

func (t *Task) SetFunc(f TaskFunc)

SetFunc sets the TaskFunc which is run when the parent command is run if this is left unset, the defaultTaskFunc will be executed instead

func (*Task) SetInitFunc

func (t *Task) SetInitFunc(f TaskFunc)

SetInitFunc sets the TaskFunc which is executed before the main TaskFunc. It's pupose is to do any setup of the DockerClient which depends on command line args for example

type TaskFunc

type TaskFunc func(t *Task, args []string)

TaskFunc is a function executed by a Task when the command the Task belongs to is run