cion

package module
v0.0.0-...-bd0def6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 12, 2015 License: Apache-2.0 Imports: 27 Imported by: 0

README

cion

A commit-to-deploy system based on Docker containers.

Build

Install Go and Node, set your GOPATH, and then run:

$ go get github.com/rohansingh/cion
$ cd $GOPATH/src/github.com/rohansingh/cion/public
$ npm install -g gulp
$ gulp

This will download cion to $GOPATH/src, build it, and install the executable to $GOPATH/bin.

The final steps are necessary to compile the resource bundle for the UI. This should be automated at some point.

Run

Currently cion will only run from its source directory:

$ cd $GOPATH/src/github.com/rohansingh/cion
$ go install ./... && cion

Note that only one instance can run at a time. Any new instances will just wait around trying to acquire a lock on the database file.

API Examples

# kick off a build of the rohan/cion branch of https://github.com/spotify/docker-client
curl -X POST http://localhost:8000/api/spotify/docker-client/branch/rohan/cion/new

# kick off a build of the master branch of cion
curl -X POST http://localhost:8000/api/rohansingh/cion/new

# get a list of all jobs for spotify/docker-client
curl -X GET http://localhost:8000/api/spotify/docker-client

# get a particular job by number
curl -X GET http://localhost:8000/api/spotify/docker-client/1

# get the logs for a particular job by number
curl -X GET http://localhost:8000/api/spotify/docker-client/1/log

User Guide

Build Steps

These are the logical steps needed to go from commit to deploy:

  1. Pull down repo and parse configuration. Based on configuration, choose which images will be used for each of the following steps.

  2. Run each service container. These containers provide services that are necessary for the build (for example, Docker).

  3. Run the build container. Perhaps in the future, build and tests will be separated and we will also run a separate test container.

  4. Run the deployment container.

We have a single system that tracks the output and state of each of these steps. It becomes the reference source for which builds were successful and what was deployed where.

Configuration Spec

All configuration is specified in .cion.yml in the project. Here's an example:

build:
  image: rohan/my-build-image

release:
  image: rohan/my-release-image
  cmd: some-optional-command

services:
  docker:
    image: jpetazzo/dind
    privileged: true
    env: # any environment variables for the service container
      - PORT=2375
    ports: # list af ports to expose from the service container
      - 2375/tcp

  some_service:
    image: rohan/some-other-image

The specified build and release containers are run for the build and release steps of the build, respectively.

Job Runner

For each commit, we generate a single job. The job runner pulls the project from Git, parses the .cion.yml configuration, and runs the service, build, and release containers.

Working directory container

Prior to running the service, build, and release containers, the job runner actually launches a data container to contain the working directory for the build. This container is also linked to the build and release containers.

Service Containers

Each service container is spawned at the beginning of the build, and is linked into the build and release containers via Docker links.

For example, this is roughly how the build container is run based on the sample configuration above:

$ docker run --link docker:docker --link some_service:some_service rohan/my-build-image

Build and test containers can use these environment variables to determine how to connect to the service containers.

Build Container

Environment variables that are passed to the build container:

  • BUILD_DIR
    The path to the working directory that contains the current commit.

  • ARTIFACTS_DIR
    The path to the artifacts directory.

  • Additional environment variables from the user's config.

  • Environment variables generated by Docker links for any service containers.

The expectation is that the build container will build the project and place generated artifacts in the ARTIFACTS_DIR.

Release Container

Environment variables that are passed to the release container:

  • ARTIFACTS_DIR
    The path to the artifacts directory.

  • Additional environment variables from the user's config.

  • Environment variables generated by Docker links for any service containers.

The expectation is that the release container will release the project and write the status to stdout/stderr.

Documentation

Index

Constants

View Source 
const (
	// GitImage is the Docker image used for the working directory and fetching sources.
	GitImage = "radial/busyboxplus:git"

	// BuildDir is the path for project sources in the working directory container.
	BuildDir = "/cion/build"

	// ArtifactsDir is the path to the artifacts directory in the working directory container.
	ArtifactsDir = "/cion/artifacts"
)

Variables

View Source 
var (
	JobsBucket    = []byte("jobs")
	JobRefsBucket = []byte("jobrefs")
)

Functions

func GetJobHandler 

func GetJobHandler(c web.C, w http.ResponseWriter, r *http.Request)

func GetLogHandler 

func GetLogHandler(c web.C, w http.ResponseWriter, r *http.Request)

func ListJobsHandler 

func ListJobsHandler(c web.C, w http.ResponseWriter, r *http.Request)

func ListOwnersHandler 

func ListOwnersHandler(c web.C, w http.ResponseWriter, r *http.Request)

func ListReposHandler 

func ListReposHandler(c web.C, w http.ResponseWriter, r *http.Request)

func NewJobHandler 

func NewJobHandler(c web.C, w http.ResponseWriter, r *http.Request)

func Run 

func Run(conf Config)

func RunLocal 

func RunLocal(path string, conf Config)

func Uint64ToBytes 

func Uint64ToBytes(x uint64) []byte

Types

type BoltJobLogger 

type BoltJobLogger struct {
	// contains filtered or unexported fields
}

func (BoltJobLogger) Write 

func (l BoltJobLogger) Write(p []byte) (int, error)

func (BoltJobLogger) WriteStep 

func (l BoltJobLogger) WriteStep(name string) error

func (BoltJobLogger) WriteTo 

func (l BoltJobLogger) WriteTo(w io.Writer) (int64, error)

type BoltJobStore 

type BoltJobStore struct {
	// contains filtered or unexported fields
}

func NewBoltJobStore 

func NewBoltJobStore(path string) (*BoltJobStore, error)

func (*BoltJobStore) GetByNumber 

func (s *BoltJobStore) GetByNumber(owner, repo string, number uint64) (*Job, error)

func (*BoltJobStore) GetLogger 

func (s *BoltJobStore) GetLogger(j *Job) JobLogger

func (*BoltJobStore) List 

func (s *BoltJobStore) List(owner, repo string) ([]*Job, error)

func (*BoltJobStore) ListOwners 

func (s *BoltJobStore) ListOwners() ([]string, error)

func (*BoltJobStore) ListRepos 

func (s *BoltJobStore) ListRepos(owner string) ([]string, error)

func (*BoltJobStore) Save 

func (s *BoltJobStore) Save(j *Job) error

Save writes job data to various buckets in the Bolt database. We use this nesting pattern for buckets: 

jobs -> (owner) -> (repo) -> logs_(job number)

The actual data for a job is saved to the bucket for its repo, and its logs are stored in the logs_<number> sub-bucket.

type Config 

type Config struct {
	Executor       Executor
	JobStore       JobStore
	GitHubClientID string
	GitHubSecret   string
}

func Configure 

func Configure(dockerEndpoint, dockerCertPath, cionDbPath, ghClientID, ghSecret string) Config

func ConfigureLocal 

func ConfigureLocal(dockerEndpoint, dockerCertPath, ghClientID, ghSecret string) Config

type ContainerConfig 

type ContainerConfig struct {
	Image      string
	Cmd        []string
	Env        []string
	Ports      []string
	Privileged bool
}

ContainerConfig is a container configuration defined in .cion.yml.

type DockerExecutor 

type DockerExecutor struct {
	// contains filtered or unexported fields
}

DockerExecutor is an Executor that runs against a single Docker host.

func NewDockerExecutor 

func NewDockerExecutor(endpoint, certPath string) (*DockerExecutor, error)

func (DockerExecutor) Attach 

func (e DockerExecutor) Attach(id string, stdout io.Writer, stderr io.Writer) error

func (DockerExecutor) Build 

func (e DockerExecutor) Build(input io.Reader, output io.Writer) (string, error)

func (DockerExecutor) Kill 

func (e DockerExecutor) Kill(id string) error

func (DockerExecutor) Run 

func (e DockerExecutor) Run(opts RunContainerOpts) (string, error)

func (DockerExecutor) Wait 

func (e DockerExecutor) Wait(id string) (int, error)

type Executor 

type Executor interface {
	// Run starts a Docker container and returns the container name if successful.
	Run(opts RunContainerOpts) (string, error)

	// Attach attaches to a container and writes stdout/stderr to the provided writers.
	Attach(id string, stdout io.Writer, stderr io.Writer) error

	// Wait blocks until a container exits, and returns its exit code.
	Wait(id string) (int, error)

	// Kill kills a container dead.
	Kill(id string) error

	// Build builds a Docker image and returns the image name if successful.
	Build(input io.Reader, output io.Writer) (string, error)
}

An Executor runs Docker containers against a Docker host or Docker-like cluster.

type InMemoryJobStore 

type InMemoryJobStore struct {
	// contains filtered or unexported fields
}

InMemoryJobStore is a mock JobStore that only stores jobs in memory and writes logs directly to stdout. It is only meant to be used for testing.

func NewInMemoryJobStore 

func NewInMemoryJobStore() *InMemoryJobStore

func (*InMemoryJobStore) GetByNumber 

func (s *InMemoryJobStore) GetByNumber(owner, repo string, number uint64) (*Job, error)

func (*InMemoryJobStore) GetLogger 

func (s *InMemoryJobStore) GetLogger(j *Job) JobLogger

func (*InMemoryJobStore) List 

func (s *InMemoryJobStore) List(owner, repo string) ([]*Job, error)

func (*InMemoryJobStore) ListOwners 

func (s *InMemoryJobStore) ListOwners() ([]string, error)

func (*InMemoryJobStore) ListRepos 

func (s *InMemoryJobStore) ListRepos(owner string) ([]string, error)

func (*InMemoryJobStore) Save 

func (s *InMemoryJobStore) Save(j *Job) error

type Job 

type Job struct {
	Number uint64

	Owner  string
	Repo   string
	Branch string
	SHA    string

	LocalPath string

	StartedAt *time.Time
	EndedAt   *time.Time

	Success bool
}

Job represents the job data that should be persisted to a JobStore.

func NewJob 

func NewJob(owner, repo, branch, sha string) *Job

type JobConfig 

type JobConfig struct {
	Build    ContainerConfig
	Release  ContainerConfig
	Services map[string]ContainerConfig
}

JobConfig is the job configuration defined by .cion.yml.

type JobLogger 

type JobLogger interface {
	io.Writer
	io.WriterTo

	// WriteStep writes a transition to a new build step to the log. All subsequent writes are
	// assumed to be part of the new build step, until another new step is written.
	WriteStep(name string) error
}

JobLogger provides an io.Writer interface for writing build logs for a job.

type JobRequest 

type JobRequest struct {
	Job      *Job
	Executor Executor
	Store    JobStore

	GitHubClientID string
	GitHubSecret   string
}

JobRequest defines a job that needs to be run and the dependencies needed to run it.

func (JobRequest) Run 

func (r JobRequest) Run()

Run executes a JobRequest and logs the results to the JobStore.

type JobStore 

type JobStore interface {
	// GetByNumber gets a job for the given owner/repo/branch by its number.
	GetByNumber(owner, repo string, number uint64) (*Job, error)

	// ListOwners returns a list of all the repo owners that have jobs.
	ListOwners() ([]string, error)

	// ListRepos returns a list of all the repos for a given owner.
	ListRepos(owner string) ([]string, error)

	// List gets all the jobs for the given owner/repo.
	List(owner, repo string) ([]*Job, error)

	// Save persists a job to storage. If the job doesn't have a number yet, it is assigned the
	// next incrementing job number for the repo.
	Save(j *Job) error

	// GetLogger gets the JobLogger to write logs for a job.
	GetLogger(j *Job) JobLogger
}

JobStore write and reads jobs and job logs from persistent storage.

type RunContainerOpts 

type RunContainerOpts struct {
	// Image is the name of the Docker image to run.
	Image string

	// Cmd is the command to run in the container.
	Cmd []string

	// Env is a list of environment variables for the container, in the form "KEY=value".
	Env []string

	// Volumes is a list of volumes to create in the container.
	Volumes []string

	// Links is a list of existing containers that should be linked into the new container,
	// in the form "container_nname:alias".
	Links []string

	// VolumesFrom is a list of existing containers whose volumes should be mounted in the new
	// container, in the form "container_name[:ro|:rw]".
	VolumesFrom []string

	// Privileged specifies whether the container has extended privileges.
	Privileged bool

	// WorkingDir is the working directory in the container.
	WorkingDir string

	// Ports is a list of container ports to expose, in the format <port>/<tcp|udp>.
	Ports []string

	// LocalImage specifies whether the image was built locally (so we shouldn't try to pull it
	// from a remote repo).
	LocalImage bool
}

RunContainerOpts are options for running a new container.

type WriterLogger 

type WriterLogger struct {
	// contains filtered or unexported fields
}

func NewWriterLogger 

func NewWriterLogger(w io.Writer) WriterLogger

func (WriterLogger) Write 

func (wl WriterLogger) Write(p []byte) (int, error)

func (WriterLogger) WriteStep 

func (wl WriterLogger) WriteStep(name string) error

func (WriterLogger) WriteTo 

func (wl WriterLogger) WriteTo(w io.Writer) (int64, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
cion

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.