sti

README

go-sti

Source-to-images (sti) is a tool for building reproducable Docker images. sti produces ready-to-run images by injecting a user source into a docker image and preparing a new Docker image which incorporates the base image and built source, and is ready to use with docker run. sti supports:

1. Incremental builds which re-use previously downloaded dependencies, previously built artifacts, etc
2. Build on one image, deploy on another with extended builds

Interested in learning more? Read on!

Philosophy

1. Simplify the process of application source + base image -> usable image for most use cases (the 80%)
2. Define and implement a workflow for incremental build that eventually uses only docker primitives
3. Develop tooling that can assist in verifying that two different base images result in the same "docker run" outcome for the same input
4. Use native docker primitives to accomplish this - map out useful improvements to docker that benefit all image builders

Anatomy of a source image

Building source images is as easy as implementing two scripts. sti expects the following scripts in /usr/bin:

1. prepare : This script is responsible for building and/or deploying the source
2. run: This script is responsible for running the deployed source

Build methodologies

sti implements two methodologies for building Docker images. The first will be familiar to anyone who's built their own Docker image before - it's just docker build. When building this way, sti generates a Dockerfile and calls docker build to produce the output image:

1. sti generates a Dockerfile to describe the output image:
   1. Based on the build image
   2. Adds the application source at /tmp/src in the container
   3. Calls /usr/bin/prepare in the container
   4. Sets the image's default command to /usr/bin/run
2. sti calls docker build to produce the output image

sti also supports building images with docker run. When building this way, the workflow is:

1. sti creates a container based on the build image. with:
   1. The application source bind-mounted to /tmp/src
   2. The build artifacts bind-mounted to /tmp/artifacts (if applicable - see incremental builds)
   3. Runs the build image's /usr/bin/prepare script
2. sti starts the container and waits for it to finish running
3. sti commits the container, setting the CMD for the output image to be /usr/bin/run

The build methodology is controlled by the -m option, and defaults to build. To build with docker run, use -m run.

Basic (--clean) builds

sti accepts the following inputs to do a build:

1. Application source: this can be source code, zipped source, a binary, etc
2. Build image: the basis for the new image to build
3. Application image tag: the tag to use for the newly created application image

The basic build process is as follows:

1. sti pulls the build image if it is not already present on the system
2. sti builds the new image from the supplied build image and source, tagging the output image with the supplied tag

Incremental builds

sti automatically detects:

1. Whether a source image is compatible with incremental building
2. Whether an incremental build can be formed when an image is compatible

If the source image is compatible, a prior build already exists, and the --clean option is not used, the workflow is as follows:

1. sti creates a new docker container from the prior build image, with a volume in /tmp/artifacts
2. sti runs /usr/bin/save-artifacts in this container - this script is responsible for copying the build artifacts into /tmp/artifacts.
3. sti builds the new output image using the selected build methodology:
   1. The artifacts from the previous build will be in /tmp/artifacts during the build
   2. The build image's /usr/bin/prepare script is responsible for detecting and using the build artifacts

Note the invocation of the save-artifacts script; this script is responsible for moving build dependencies to /tmp/artifacts

Extended builds

Extended builds allow you to execute your build on a build image, then deploy it on a different runtime image. The workflow for extended builds is as follows:

1. sti looks for the previous build image for the tag, <tag>-build.
2. If that image exists:
   1. sti creates a container from this image and runs /usr/bin/save-artifacts in it
3. sti creates a build container from the build image with a volume at /tmp/build and bind-mounts in the artifacts from the prior build, if applicable
4. sti runs /usr/bin/prepare in the build container - this script is responsible for populating /tmp/build with the result of the build
5. sti builds the output image with the selected build methodology:
   1. The base image will be the runtime image
   2. The output of the source build step will be in /tmp/src during the build
   3. The runtime image's /usr/bin/prepare script is responsible for being able to deploy the artifact in /tmp/src
6. If the docker build succeeds, the build container is tagged as <tag>-build

You might have noticed that the above workflow describes something like an incremental build. This behavior can be disabled with the --clean option.

Getting started

Dependencies

1. Docker
2. Go

Installation

go get github.com/openshift/geard/pkg/go-sti/sti

Example

You can start using sti right away with the following test sources and publicly available images:

sti build git://github.com/pmorie/simple-ruby pmorie/centos-ruby2 test-ruby-app
docker run -rm -i -p :9292 -t test-ruby-app

sti build git://github.com/pmorie/simple-ruby pmorie/ubuntu-buildpack test-foreman-app \
-e 'BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-ruby.git'
docker run -rm -i -p :5000 -t test-foreman-app

sti build git://github.com/pmorie/simple-html pmorie/fedora-mock test-html-app
docker run -rm -i -p :8080 -t sti_app

Validating a source image

sti validate BUILD_IMAGE_TAG [flags]

Available Flags:
     --debug=false: Enable debugging output
 -I, --incremental=false: Validate for an incremental build
 -R, --runtime="": Set the runtime image to use
 -U, --url="unix:///var/run/docker.sock": Set the url of the docker socket to use

You can validate that an image is usable as a sti source image as follows:

sti validate BUILD_IMAGE_TAG

The --incremental option to enables validation for incremental builds:

sti validate BUILD_IMAGE_TAG --incremental

Add the -R option to additionally validate a runtime image for extended builds:

sti validate BUILD_IMAGE_TAG -R RUNTIME_IMAGE_TAG

When specifying a runtime image with sti validate, the build image is automatically validated for incremental builds.

Building a deployable image with sti

sti build SOURCE BUILD_IMAGE APP_IMAGE_TAG [flags]

Available Flags:
     --clean=false: Perform a clean build
     --debug=false: Enable debugging output
     --dir="tempdir": Directory where generated Dockerfiles and other support scripts are created
 -e, --env="": Specify an environment var NAME=VALUE,NAME2=VALUE2,...
 -R, --runtime="": Set the runtime image to use
 -U, --url="unix:///var/run/docker.sock": Set the url of the docker socket to use

The most basic sti build uses a single build image:

sti build SOURCE BUILD_IMAGE_TAG APP_IMAGE_TAG

If the build is successful, the built image will be tagged with APP_IMAGE_TAG.

If the build image is compatible with incremental builds, sti build will look for an image tagged with APP_IMAGE_TAG. If an image is present with that tag, sti build will save the build artifacts from that image and add them to the build container at /tmp/artifacts so an image's /usr/bin/prepare script can restore them before building the source.

When using an image that supports incremental builds, you can do a clean build with --clean:

sti build SOURCE BUILD_IMAGE_TAG APP_IMAGE_TAG --clean

Extended builds allow you to use distinct images for building your sources and deploying them. Use the -R option perform an extended build targeting a runtime image:

sti build SOURCE BUILD_IMAGE_TAG APP_IMAGE_TAG -R RUNTIME_IMAGE_TAG

When specifying a runtime image, the build image must be compatible with incremental builds. sti build will look for an image tagged with <APP_IMAGE_TAG>-build. If an image is present with that tag, sti build will save the build artifacts from that image and add them to the build container at /tmp/artifacts so the build image's /usr/bin/prepare script can restore them before building the source. The build image's /usr/bin/prepare script is responsible for populating /tmp/build with an artifact to be deployed into the runtime container.

After performing the build, a new runtime image is created based on the image tagged with RUNTIME_IMAGE_TAG with the output of the build in /tmp/src. The runtime image's /usr/bin/prepare script is responsible for detecting and deploying the artifact. If the build is successful, two images are tagged:

1. The build image is tagged with <APP_IMAGE_TAG>-build
2. The prepared image incorporating the deployed build is tagged with APP_IMAGE_TAG

You can do a clean extended build with --clean:

sti build SOURCE_DIR BUILD_IMAGE_TAG APP_IMAGE_TAG -R RUNTIME_IMAGE_TAG --clean

Documentation

Index

• func FileExistsInContainer(dockerClient *docker.Client, cId string, path string) bool
• type BuildRequest
• type BuildResult
  • func Build(req BuildRequest) (*BuildResult, error)
• type Request
• type STIResult
• type StiError
  • func (s StiError) Error() string
• type ValidateRequest
• type ValidateResult
  • func Validate(req ValidateRequest) (*ValidateResult, error)

Functions

func FileExistsInContainer

func FileExistsInContainer(dockerClient *docker.Client, cId string, path string) bool

Determine whether a file exists in a container.

Types

type BuildRequest

type BuildRequest struct {
	Request
	Source      string
	Tag         string
	Clean       bool
	Environment map[string]string
	Method      string
	Writer      io.Writer
}

type BuildResult

type BuildResult STIResult

func Build

func Build(req BuildRequest) (*BuildResult, error)

Build processes a BuildRequest and returns a *BuildResult and an error. An error represents a failure performing the build rather than a failure of the build itself. Callers should check the Success field of the result to determine whether a build succeeded or not.

type Request

type Request struct {
	BaseImage    string
	RuntimeImage string

	DockerSocket  string
	DockerTimeout int
	WorkingDir    string
	Debug         bool
}

Request contains essential fields for any request: a Configuration, a base image, and an optional runtime image.

type STIResult

type STIResult struct {
	Success  bool
	Messages []string
}

type StiError

type StiError int

const (
	ErrDockerConnectionFailed StiError = iota
	ErrNoSuchBaseImage
	ErrNoSuchRuntimeImage
	ErrPullImageFailed
	ErrSaveArtifactsFailed
	ErrCreateDockerfileFailed
	ErrCreateContainerFailed
	ErrInvalidBuildMethod
	ErrBuildFailed
	ErrCommitContainerFailed
)

func (StiError) Error

func (s StiError) Error() string

type ValidateRequest

type ValidateRequest struct {
	Request
	Incremental bool
}

Describes a request to validate an images for use in an sti build.

type ValidateResult

type ValidateResult STIResult

func Validate

func Validate(req ValidateRequest) (*ValidateResult, error)

Service the supplied ValidateRequest and return a ValidateResult.