imagebuilder

README

OCI Image Builder

Please test your images (and add to our conformance suite)!

This library supports using the Dockerfile syntax to build OCI & Docker compatible images, without invoking a container build command such as buildah bud or docker build. It is intended to give clients more control over how they build container images, including:

• Instead of building one layer per line, run all instructions in the same container
• Set HostConfig settings like network and memory controls that are not available when running container builds
• Mount external files into the build that are not persisted as part of the final image (i.e. "secrets")
• If there are no RUN commands in the Dockerfile, the container is created and committed, but never started.

The final image should be 99.9% compatible with regular container builds, but bugs are always possible.

Future goals include:

• Output OCI compatible images
• Support other container execution engines, like runc or rkt
• Better conformance testing
• Windows support

Install and Run

To download and install the library and the binary, set up a Golang build environment and with GOPATH set run:

$ go get -u github.com/openshift/imagebuilder/cmd/imagebuilder

The included command line takes one argument, a path to a directory containing a Dockerfile. The -t option can be used to specify an image to tag as:

$ imagebuilder [-t TAG] DIRECTORY

To mount a file into the image for build that will not be present in the final output image, run:

$ imagebuilder --mount ~/secrets/private.key:/etc/keys/private.key path/to/my/code testimage

Any processes in the Dockerfile will have access to /etc/keys/private.key, but that file will not be part of the committed image.

You can also customize which Dockerfile is run, or run multiple Dockerfiles in sequence (the FROM is ignored on later files):

$ imagebuilder -f Dockerfile:Dockerfile.extra .

will build the current directory and combine the first Dockerfile with the second. The FROM in the second image is ignored.

Note that imagebuilder adds the built image to the docker daemon's internal storage. If you use podman you must first pull the image into its local registry:

$ podman pull docker-daemon:<IMAGE>:<TAG> # must contain either a tag or a digest

Code Example

f, err := os.Open("path/to/Dockerfile")
if err != nil {
	return err
}
defer f.Close()

e := builder.NewClientExecutor(o.Client)
e.Out, e.ErrOut = os.Stdout, os.Stderr
e.AllowPull = true
e.Directory = "context/directory"
e.Tag = "name/of-image:and-tag"
e.AuthFn = nil // ... pass a function to retrieve authorization info
e.LogFn = func(format string, args ...interface{}) {
	fmt.Fprintf(e.ErrOut, "--> %s\n", fmt.Sprintf(format, args...))
}

buildErr := e.Build(f, map[string]string{"arg1":"value1"})
if err := e.Cleanup(); err != nil {
	fmt.Fprintf(e.ErrOut, "error: Unable to clean up build: %v\n", err)
}

return buildErr

Example of usage from OpenShift's experimental dockerbuild command with mount secrets

Run conformance tests (very slow):

docker rmi busybox; docker pull busybox
docker rmi centos:7; docker pull centos:7
chmod -R go-w ./dockerclient/testdata
go test ./dockerclient -tags conformance -timeout 30m

Documentation

Overview

Package builder uses code from github.com/docker/docker/builder/* to implement a Docker builder that does not create individual layers, but instead creates a single layer.

TODO: full windows support

Index

• Constants
• Variables
• func BashQuote(env string) string
• func ExportEnv(env []string) string
• func ParseDockerfile(r io.Reader) (*parser.Node, error)
• func ParseDockerignore(root string) ([]string, error)
• func ParseFile(path string) (*parser.Node, error)
• func ParseIgnore(path string) ([]string, error)
• func ProcessWord(word string, env []string) (string, error)
• func ProcessWords(word string, env []string) ([]string, error)
• func SplitBy(node *parser.Node, value string) []*parser.Node
• func SplitChildren(node *parser.Node, value string) []*parser.Node
• type Builder
  • func NewBuilder(args map[string]string) *Builder
  • func (b *Builder) Arguments() []string
  • func (b *Builder) Config() *docker.Config
  • func (b *Builder) From(node *parser.Node) (string, error)
  • func (b *Builder) FromImage(image *docker.Image, node *parser.Node) error
  • func (b *Builder) RequiresStart(node *parser.Node) bool
  • func (b *Builder) Run(step *Step, exec Executor, noRunsRemaining bool) error
  • func (b *Builder) Step() *Step
• type Copy
• type Executor
• type Run
• type Stage
• type Stages
  • func NewStages(node *parser.Node, b *Builder) (Stages, error)
  • func (stages Stages) ByName(name string) (Stage, bool)
  • func (stages Stages) ByTarget(target string) (Stages, bool)
  • func (stages Stages) ThroughTarget(target string) (Stages, bool)
• type Step
  • func (b *Step) Resolve(ast *parser.Node) error
• type StepFunc
• type VolumeSet
  • func (s *VolumeSet) Add(path string) bool
  • func (s VolumeSet) Covers(path string) bool
  • func (s VolumeSet) Has(path string) bool

Constants

const (
	// in docker/system
	NoBaseImageSpecifier = "scratch"
)

Variables

var (
	LogExecutor  = logExecutor{}
	NoopExecutor = noopExecutor{}
)

var ErrNoFROM = fmt.Errorf("no FROM statement found")

ErrNoFROM is returned if the Dockerfile did not contain a FROM statement.

Functions

func BashQuote

func BashQuote(env string) string

BashQuote escapes the provided string and surrounds it with double quotes. TODO: verify that these are all we have to escape.

func ExportEnv

func ExportEnv(env []string) string

ExportEnv creates an export statement for a shell that contains all of the provided environment.

func ParseDockerfile

func ParseDockerfile(r io.Reader) (*parser.Node, error)

ParseDockerfile parses the provided stream as a canonical Dockerfile

func ParseDockerignore

func ParseDockerignore(root string) ([]string, error)

ParseDockerIgnore returns a list of the excludes in the .containerignore or .dockerignore file.

func ParseFile

func ParseFile(path string) (*parser.Node, error)

func ParseIgnore

func ParseIgnore(path string) ([]string, error)

ParseIgnore returns a list of the excludes in the specified path path should be a file with the .dockerignore format extracted from fsouza/go-dockerclient and modified to drop comments and empty lines.

func ProcessWord

func ProcessWord(word string, env []string) (string, error)

ProcessWord will use the 'env' list of environment variables, and replace any env var references in 'word'.

func ProcessWords

func ProcessWords(word string, env []string) ([]string, error)

ProcessWords will use the 'env' list of environment variables, and replace any env var references in 'word' then it will also return a slice of strings which represents the 'word' split up based on spaces - taking into account quotes. Note that this splitting is done **after** the env var substitutions are done. Note, each one is trimmed to remove leading and trailing spaces (unless they are quoted", but ProcessWord retains spaces between words.

func SplitBy

func SplitBy(node *parser.Node, value string) []*parser.Node

func SplitChildren

func SplitChildren(node *parser.Node, value string) []*parser.Node

SplitChildren removes any children with the provided value from node and returns them as an array. node.Children is updated.

Types

type Builder

type Builder struct {
	RunConfig docker.Config

	Env         []string
	Args        map[string]string
	HeadingArgs map[string]string
	UserArgs    map[string]string
	CmdSet      bool
	Author      string
	// Certain instructions like `FROM` will need to use
	// `ARG` decalred before or not in this stage hence
	// while processing instruction like `FROM ${SOME_ARG}`
	// we will make sure to verify if they are declared any
	// where in containerfile or not.
	GlobalAllowedArgs []string

	AllowedArgs map[string]bool
	Volumes     VolumeSet
	Excludes    []string

	PendingVolumes VolumeSet
	PendingRuns    []Run
	PendingCopies  []Copy

	Warnings []string
	// Raw platform string specified with `FROM --platform` of the stage
	// It's up to the implementation or client to parse and use this field
	Platform string
}

func NewBuilder

func NewBuilder(args map[string]string) *Builder

func (*Builder) Arguments

func (b *Builder) Arguments() []string

Arguments returns the currently active arguments.

func (*Builder) Config

func (b *Builder) Config() *docker.Config

Config returns a snapshot of the current RunConfig intended for use with a container commit.

func (*Builder) From

func (b *Builder) From(node *parser.Node) (string, error)

From returns the image this dockerfile depends on, or an error if no FROM is found or if multiple FROM are specified. If a single from is found the passed node is updated with only the remaining statements. The builder's RunConfig.Image field is set to the first From found, or left unchanged if already set.

func (*Builder) FromImage

func (b *Builder) FromImage(image *docker.Image, node *parser.Node) error

FromImage updates the builder to use the provided image (resetting RunConfig and recording the image environment), and updates the node with any ONBUILD statements extracted from the parent image.

func (*Builder) RequiresStart

func (b *Builder) RequiresStart(node *parser.Node) bool

RequiresStart returns true if a running container environment is necessary to invoke the provided commands

func (*Builder) Run

func (b *Builder) Run(step *Step, exec Executor, noRunsRemaining bool) error

Run executes a step, transforming the current builder and invoking any Copy or Run operations. noRunsRemaining is an optimization hint that allows the builder to avoid performing unnecessary work.

func (*Builder) Step

func (b *Builder) Step() *Step

Step creates a new step from the current state.

type Copy

type Copy struct {
	// If true, this is a copy from the file system to the container. If false,
	// the copy is from the context.
	FromFS bool
	// If set, this is a copy from the named stage or image to the container.
	From     string
	Src      []string
	Dest     string
	Download bool
	// If set, the owner:group for the destination.  This value is passed
	// to the executor for handling.
	Chown string
	Chmod string
}

Copy defines a copy operation required on the container.

type Executor

type Executor interface {
	Preserve(path string) error
	// EnsureContainerPath should ensure that the directory exists, creating any components required
	EnsureContainerPath(path string) error
	// EnsureContainerPathAs should ensure that the directory exists, creating any components required
	// with the specified owner and mode, if either is specified
	EnsureContainerPathAs(path, user string, mode *os.FileMode) error
	Copy(excludes []string, copies ...Copy) error
	Run(run Run, config docker.Config) error
	UnrecognizedInstruction(step *Step) error
}

type Run

type Run struct {
	Shell bool
	Args  []string
	// Mounts are mounts specified through the --mount flag inside the Containerfile
	Mounts []string
	// Network specifies the network mode to run the container with
	Network string
}

Run defines a run operation required in the container.

type Stage

type Stage struct {
	Position int
	Name     string
	Builder  *Builder
	Node     *parser.Node
}

type Stages

type Stages []Stage

func NewStages

func NewStages(node *parser.Node, b *Builder) (Stages, error)

func (Stages) ByName

func (stages Stages) ByName(name string) (Stage, bool)

func (Stages) ByTarget

func (stages Stages) ByTarget(target string) (Stages, bool)

Get just the target stage.

func (Stages) ThroughTarget

func (stages Stages) ThroughTarget(target string) (Stages, bool)

Get all the stages up to and including the target.

type Step

type Step struct {
	Env []string

	Command  string
	Args     []string
	Flags    []string
	Attrs    map[string]bool
	Message  string
	Original string
}

Step represents the input Env and the output command after all post processing of the command arguments is done.

func (*Step) Resolve

func (b *Step) Resolve(ast *parser.Node) error

Resolve transforms a parsed Dockerfile line into a command to execute, resolving any arguments.

Almost all nodes will have this structure: Child[Node, Node, Node] where Child is from parser.Node.Children and each node comes from parser.Node.Next. This forms a "line" with a statement and arguments and we process them in this normalized form by hitting evaluateTable with the leaf nodes of the command and the Builder object.

ONBUILD is a special case; in this case the parser will emit: Child[Node, Child[Node, Node...]] where the first node is the literal "onbuild" and the child entrypoint is the command of the ONBUILD statement, such as `RUN` in ONBUILD RUN foo. There is special case logic in here to deal with that, at least until it becomes more of a general concern with new features.

type StepFunc

type StepFunc func(*Builder, []string, map[string]bool, []string, string) error

StepFunc is invoked with the result of a resolved step.

type VolumeSet

type VolumeSet []string

func (*VolumeSet) Add

func (s *VolumeSet) Add(path string) bool

func (VolumeSet) Covers

func (s VolumeSet) Covers(path string) bool

func (VolumeSet) Has

func (s VolumeSet) Has(path string) bool