imagebuilder

README

OCI Image Builder

Please test your images (and add to our conformance suite)!

This library supports using the Dockerfile syntax to build OCI & Docker compatible images, without invoking a container build command such as buildah bud or docker build. It is intended to give clients more control over how they build container images, including:

• Instead of building one layer per line, run all instructions in the same container
• Set HostConfig settings like network and memory controls that are not available when running container builds
• Mount external files into the build that are not persisted as part of the final image (i.e. "secrets")
• If there are no RUN commands in the Dockerfile, the container is created and committed, but never started.

The final image should be 99.9% compatible with regular container builds, but bugs are always possible.

Future goals include:

• Output OCI compatible images
• Support other container execution engines, like runc or rkt
• Better conformance testing
• Windows support

Install and Run

To download and install the library and the binary, set up a Golang build environment and with GOPATH set run:

$ go install github.com/openshift/imagebuilder/cmd/imagebuilder@latest

The included command line takes one argument, a path to a directory containing a Dockerfile. The -t option can be used to specify an image to tag as:

$ imagebuilder [-t TAG] DIRECTORY

To mount a file into the image for build that will not be present in the final output image, run:

$ imagebuilder --mount ~/secrets/private.key:/etc/keys/private.key path/to/my/code testimage

Any processes in the Dockerfile will have access to /etc/keys/private.key, but that file will not be part of the committed image.

You can also customize which Dockerfile is run, or run multiple Dockerfiles in sequence (the FROM is ignored on later files):

$ imagebuilder -f Dockerfile:Dockerfile.extra .

will build the current directory and combine the first Dockerfile with the second. The FROM in the second image is ignored.

Note that imagebuilder adds the built image to the docker daemon's internal storage. If you use podman you must first pull the image into its local registry:

$ podman pull docker-daemon:<IMAGE>:<TAG> # must contain either a tag or a digest

Code Example

f, err := os.Open("path/to/Dockerfile")
if err != nil {
	return err
}
defer f.Close()

e := builder.NewClientExecutor(o.Client)
e.Out, e.ErrOut = os.Stdout, os.Stderr
e.AllowPull = true
e.Directory = "context/directory"
e.Tag = "name/of-image:and-tag"
e.AuthFn = nil // ... pass a function to retrieve authorization info
e.LogFn = func(format string, args ...interface{}) {
	fmt.Fprintf(e.ErrOut, "--> %s\n", fmt.Sprintf(format, args...))
}

buildErr := e.Build(f, map[string]string{"arg1":"value1"})
if err := e.Cleanup(); err != nil {
	fmt.Fprintf(e.ErrOut, "error: Unable to clean up build: %v\n", err)
}

return buildErr

Example of usage from OpenShift's experimental dockerbuild command with mount secrets

Run conformance tests (very slow):

docker rmi busybox; docker pull busybox
docker rmi alpine; docker pull alpine
docker rmi centos:7; docker pull centos:7
docker rmi registry.fedoraproject.org/fedora-minimal; docker pull registry.fedoraproject.org/fedora-minimal
docker rmi registry.fedoraproject.org/fedora-minimal:41-x86_64; docker pull registry.fedoraproject.org/fedora-minimal:41-x86_64
docker rmi registry.fedoraproject.org/fedora-minimal:41-aarch64; docker pull registry.fedoraproject.org/fedora-minimal:41-aarch64
chmod -R go-w ./dockerclient/testdata
go test ./dockerclient -tags conformance -timeout 30m

Documentation

Overview

Package builder uses code from github.com/docker/docker/builder/* to implement a Docker builder that does not create individual layers, but instead creates a single layer.

TODO: full windows support

Index

• Constants
• Variables
• func BashQuote(env string) string
• func ExportEnv(env []string) string
• func ParseDockerfile(r io.Reader) (*parser.Node, error)
• func ParseDockerignore(root string) ([]string, error)
• func ParseFile(path string) (*parser.Node, error)
• func ParseIgnore(path string) ([]string, error)
• func ProcessWord(word string, env []string) (string, error)
• func ProcessWords(word string, env []string) ([]string, error)
• func SplitBy(node *parser.Node, value string) []*parser.Node
• func SplitChildren(node *parser.Node, value string) []*parser.Node
• type Builder
  • func NewBuilder(args map[string]string) *Builder
  • func (b *Builder) Arguments() []string
  • func (b *Builder) Config() *docker.Config
  • func (b *Builder) From(node *parser.Node) (string, error)
  • func (b *Builder) FromImage(image *docker.Image, node *parser.Node) error
  • func (b *Builder) RequiresStart(node *parser.Node) bool
  • func (b *Builder) Run(step *Step, exec Executor, noRunsRemaining bool) error
  • func (b *Builder) Step() *Step
• type Copy
• type Executor
• type File
• type Run
• type Stage
• type Stages
  • func NewStages(node *parser.Node, b *Builder) (Stages, error)
  • func (stages Stages) ByName(name string) (Stage, bool)
  • func (stages Stages) ByPosition(position int) (Stage, bool)
  • func (stages Stages) ByTarget(target string) (Stages, bool)
  • func (stages Stages) ThroughTarget(target string) (Stages, bool)
• type Step
  • func (b *Step) Resolve(ast *parser.Node) error
• type StepFunc
• type VolumeSet
  • func (s *VolumeSet) Add(path string) bool
  • func (s VolumeSet) Covers(path string) bool
  • func (s VolumeSet) Has(path string) bool

Constants

const (
	// in docker/system
	NoBaseImageSpecifier = "scratch"
)

Variables

var (
	LogExecutor  = logExecutor{}
	NoopExecutor = noopExecutor{}
)

var ErrNoFROM = fmt.Errorf("no FROM statement found")

ErrNoFROM is returned if the Dockerfile did not contain a FROM statement.

Functions

func BashQuote

func BashQuote(env string) string

BashQuote escapes the provided string and surrounds it with double quotes. TODO: verify that these are all we have to escape.

func ExportEnv

func ExportEnv(env []string) string

ExportEnv creates an export statement for a shell that contains all of the provided environment.

func ParseDockerfile

func ParseDockerfile(r io.Reader) (*parser.Node, error)

ParseDockerfile parses the provided stream as a canonical Dockerfile

func ParseDockerignore

func ParseDockerignore(root string) ([]string, error)

ParseDockerIgnore returns a list of the excludes in the .containerignore or .dockerignore file.

func ParseFile

func ParseFile(path string) (*parser.Node, error)

func ParseIgnore

func ParseIgnore(path string) ([]string, error)

ParseIgnore returns a list of the excludes in the specified path path should be a file with the .dockerignore format extracted from fsouza/go-dockerclient and modified to drop comments and empty lines.

func ProcessWord

func ProcessWord(word string, env []string) (string, error)

ProcessWord will use the 'env' list of environment variables, and replace any env var references in 'word'.

func ProcessWords

func ProcessWords(word string, env []string) ([]string, error)

ProcessWords will use the 'env' list of environment variables, and replace any env var references in 'word' then it will also return a slice of strings which represents the 'word' split up based on spaces - taking into account quotes. Note that this splitting is done **after** the env var substitutions are done. Note, each one is trimmed to remove leading and trailing spaces (unless they are quoted", but ProcessWord retains spaces between words.

func SplitBy

func SplitBy(node *parser.Node, value string) []*parser.Node

func SplitChildren

func SplitChildren(node *parser.Node, value string) []*parser.Node

SplitChildren removes any children with the provided value from node and returns them as an array. node.Children is updated.

Types

type Builder

type Builder struct {
	RunConfig docker.Config

	Env []string

	// Args contains values originally given to NewBuilder() or set due to
	// ARG instructions in a stage, either with a default value provided,
	// or with a default inherited from an ARG instruction in the header
	Args map[string]string
	// HeadingArgs contains the values for ARG instructions in the
	// Dockerfile which occurred before the first FROM instruction, either
	// with a default value provided as part of the ARG instruction, or
	// expecting a value to be supplied in UserArgs via NewBuilder().
	HeadingArgs map[string]string
	// UserArgs includes a copy of the values that were passed to
	// NewBuilder(), unmodified.
	UserArgs map[string]string

	CmdSet bool
	Author string

	// GlobalAllowedArgs are args which should be resolvable in a FROM
	// instruction, either built-in and always available, or introduced by
	// an ARG instruction in the header.
	GlobalAllowedArgs []string
	// AllowedArgs are args which should be resolvable in this stage,
	// having been introduced by a previous ARG instruction in this stage.
	AllowedArgs map[string]bool

	Volumes  VolumeSet
	Excludes []string

	PendingVolumes VolumeSet
	PendingRuns    []Run
	PendingCopies  []Copy

	Warnings []string
	// Raw platform string specified with `FROM --platform` of the stage
	// It's up to the implementation or client to parse and use this field
	Platform string

	// Overrides for TARGET... and BUILD... values. TARGET... values are
	// typically only necessary if the builder's target platform is not the
	// same as the build platform.
	BuiltinArgDefaults map[string]string
}

func NewBuilder

func NewBuilder(args map[string]string) *Builder

func (*Builder) Arguments

func (b *Builder) Arguments() []string

Arguments returns the currently active arguments.

func (*Builder) Config

func (b *Builder) Config() *docker.Config

Config returns a snapshot of the current RunConfig intended for use with a container commit.

func (*Builder) From

func (b *Builder) From(node *parser.Node) (string, error)

From returns the image this dockerfile depends on, or an error if no FROM is found or if multiple FROM are specified. If a single from is found the passed node is updated with only the remaining statements. The builder's RunConfig.Image field is set to the first From found, or left unchanged if already set.

func (*Builder) FromImage

func (b *Builder) FromImage(image *docker.Image, node *parser.Node) error

FromImage updates the builder to use the provided image (resetting RunConfig and recording the image environment), and updates the node with any ONBUILD statements extracted from the parent image.

func (*Builder) RequiresStart

func (b *Builder) RequiresStart(node *parser.Node) bool

RequiresStart returns true if a running container environment is necessary to invoke the provided commands

func (*Builder) Run

func (b *Builder) Run(step *Step, exec Executor, noRunsRemaining bool) error

Run executes a step, transforming the current builder and invoking any Copy or Run operations. noRunsRemaining is an optimization hint that allows the builder to avoid performing unnecessary work.

func (*Builder) Step

func (b *Builder) Step() *Step

Step creates a new step from the current state.

type Copy

type Copy struct {
	// If true, this is a copy from the file system to the container. If false,
	// the copy is from the context.
	FromFS bool
	// If set, this is a copy from the named stage or image to the container.
	From     string
	Src      []string
	Dest     string
	Download bool
	// If set, the owner:group for the destination.  This value is passed
	// to the executor for handling.
	Chown string
	Chmod string
	// If set, a checksum which the source must match, or be rejected.
	Checksum string
	// Additional files which need to be created by executor for this
	// instruction.
	Files []File
	// If set, when the source is a URL for a remote Git repository,
	// refrain from stripping out the .git subdirectory after cloning it.
	KeepGitDir bool
	// If set, instead of adding these items to the rootfs and picking them
	// up as part of a subsequent diff generation, build an archive of them
	// and include it as an independent layer.
	Link bool
	// If set, preserve leading directories in the paths of items being
	// copied, relative to either the top of the build context, or to the
	// "pivot point", a location in the source path marked by a path
	// component named "." (i.e., where "/./" occurs in the path).
	Parents bool
	// Exclusion patterns, a la .dockerignore, relative to either the top
	// of a directory tree being copied, or the "pivot point", a location
	// in the source path marked by a path component named ".".
	Excludes []string
}

Copy defines a copy operation required on the container.

type Executor

type Executor interface {
	Preserve(path string) error
	// EnsureContainerPath should ensure that the directory exists, creating any components required
	EnsureContainerPath(path string) error
	// EnsureContainerPathAs should ensure that the directory exists, creating any components required
	// with the specified owner and mode, if either is specified
	EnsureContainerPathAs(path, user string, mode *os.FileMode) error
	Copy(excludes []string, copies ...Copy) error
	Run(run Run, config docker.Config) error
	UnrecognizedInstruction(step *Step) error
}

type File

type File struct {
	Name string // Name of the new file.
	Data string // Content of the file.
}

File defines if any additional file needs to be created by the executor instruction so that specified command can execute/copy the created file inside the build container.

type Run

type Run struct {
	Shell bool
	Args  []string
	// Mounts are mounts specified through the --mount flag inside the Containerfile
	Mounts []string
	// Network specifies the network mode to run the container with
	Network string
	// Additional files which need to be created by executor for this
	// instruction.
	Files []File
}

Run defines a run operation required in the container.

type Stage

type Stage struct {
	Position int
	Name     string
	Builder  *Builder
	Node     *parser.Node
}

type Stages

type Stages []Stage

func NewStages

func NewStages(node *parser.Node, b *Builder) (Stages, error)

func (Stages) ByName

func (stages Stages) ByName(name string) (Stage, bool)

func (Stages) ByPosition

func (stages Stages) ByPosition(position int) (Stage, bool)

func (Stages) ByTarget

func (stages Stages) ByTarget(target string) (Stages, bool)

Get just the target stage.

func (Stages) ThroughTarget

func (stages Stages) ThroughTarget(target string) (Stages, bool)

Get all the stages up to and including the target.

type Step

type Step struct {
	Env []string

	Command  string
	Args     []string
	Flags    []string
	Attrs    map[string]bool
	Message  string
	Heredocs []buildkitparser.Heredoc
	Original string
}

Step represents the input Env and the output command after all post processing of the command arguments is done.

func (*Step) Resolve

func (b *Step) Resolve(ast *parser.Node) error

Resolve transforms a parsed Dockerfile line into a command to execute, resolving any arguments.

Almost all nodes will have this structure: Child[Node, Node, Node] where Child is from parser.Node.Children and each node comes from parser.Node.Next. This forms a "line" with a statement and arguments and we process them in this normalized form by hitting evaluateTable with the leaf nodes of the command and the Builder object.

ONBUILD is a special case; in this case the parser will emit: Child[Node, Child[Node, Node...]] where the first node is the literal "onbuild" and the child entrypoint is the command of the ONBUILD statement, such as `RUN` in ONBUILD RUN foo. There is special case logic in here to deal with that, at least until it becomes more of a general concern with new features.

type StepFunc

type StepFunc func(*Builder, []string, map[string]bool, []string, string, []buildkitparser.Heredoc) error

StepFunc is invoked with the result of a resolved step.

type VolumeSet

type VolumeSet []string

func (*VolumeSet) Add

func (s *VolumeSet) Add(path string) bool

func (VolumeSet) Covers

func (s VolumeSet) Covers(path string) bool

func (VolumeSet) Has

func (s VolumeSet) Has(path string) bool