blubber

README

Blubber is a highly opinionated abstraction for container build configurations and a command-line compiler which currently supports outputting multi-stage Dockerfiles. It aims to provide a handful of declarative constructs that accomplish build configuration in a more secure and determinate way than running ad-hoc commands.

Example configuration

version: v4
base: debian:jessie
apt:
  packages: [libjpeg, libyaml]
lives:
  in: /srv/service
runs:
  environment:
    FOO: bar
    BAR: baz

variants:
  build:
    apt:
      packages: [libjpeg-dev, libyaml-dev]
    node:
      requirements: [package.json, package-lock.json]
    copies: [local]

  development:
    includes: [build]

  test:
    includes: [build]
    apt:
      packages: [chromium]
    entrypoint: [npm, test]

  prep:
    includes: [build]
    node:
      env: production

  production:
    base: debian:jessie-slim
    node:
      env: production
    copies: [prep]
    entrypoint: [node, server.js]

Variants

Blubber supports a concept of composeable configuration variants for defining slightly different container images while still maintaining a sufficient degree of parity between them. For example, images for development and testing may require some development and debugging packages which you wouldn't want in production lest they contain vulnerabilities and somehow end up linked or included in the application runtime.

Properties declared at the top level are shared among all variants unless redefined, and one variant can include the properties of others. Some properties, like apt:packages are combined when inherited or included.

In the example configuration, the test variant when expanded effectively becomes:

version: v4
base: debian:jessie
apt:
  packages: [libjpeg, libyaml, libjpeg-dev, libyaml-dev, chromium]
node:
  dependencies: true
runs:
  in: /srv/service
  as: runuser
  uid: 666
  gid: 666
entrypoint: [npm, test]

Artifacts

When trying to ensure optimally sized Docker images for production, there's a common pattern that has emerged which is essentially to use one image for building an application and copying the resulting build artifacts to another much more optimized image, using the latter for production.

The Docker community has responded to this need by implementing multi-stage builds and Blubber makes use of this with its copies configuration property.

In the example configuration, the production variant declares artifacts to be copied over from the result of building the prep image.

Builders key

As an alternative to specifying the various builder keys (node, python, php and builder), it is possible to group builders in a list under the builders key. This offers two advantages:

• It defines an order of execution for the builders. Associated instructions will be generated in the order in which the builders appear in the file
• It makes it possible to specify multiple custom builders. In this case, the builder key is replaced by custom

Similarly to other configuration keys, builders appearing at the top level of the file will be applied to all variant configurations. Builder keys appearing both at the top level and in a variant, will be merged; whereas builders present only at the top level will be placed first in the execution order.

For a particular variant, builders and the standalone builder keys are mutually exclusive, but different styles can be used for different variants. However, note that top level definitions are applied to all variants, so using one style at the top level precludes the use of the other for all variants.

The example configuration rewritten to use builders becomes:

version: v4
base: debian:jessie
apt:
  packages: [libjpeg, libyaml]
lives:
  in: /srv/service
runs:
  environment:
    FOO: bar
    BAR: baz

variants:
  build:
    apt:
      packages: [libjpeg-dev, libyaml-dev]
    builders:
      - node:
          requirements: [package.json, package-lock.json]
    copies: [local]

  development:
    includes: [build]

  test:
    includes: [build]
    apt:
      packages: [chromium]
    entrypoint: [npm, test]

  prep:
    includes: [build]
    builders:
      - node:
          env: production

  production:
    base: debian:jessie-slim
    builders:
      - node:
          env: production
    copies: [prep]
    entrypoint: [node, server.js]

See file examples/blubber.builders.yaml for a more detailed example with multiple builders.

Usage

Running the blubber command will be produce Dockerfile output for the given variant.

blubber config.yaml variant

You can see the result of the example configuration by cloning this repo and running (assuming you have go):

make
./blubber examples/blubber.yaml development
./blubber examples/blubber.yaml test
./blubber examples/blubber.yaml production

Other examples with different variants can be found under directory examples.

Contribution

If you'd like to make code contributions to Blubber, see CONTRIBUTING.md.

BuildKit frontend for buildctl and docker build

In addition to a CLI and a microservice, Blubber includes a BuildKit gRPC gateway that works with both BuildKit's buildctl command and with docker build.

To build from Blubber configuration using buildctl, do:

buildctl build --frontend gateway.v0 \
  --opt source=docker-registry.wikimedia.org/wikimedia/blubber-buildkit:v0.11.1 \
  --local context=. \
  --local dockerfile=. \
  --opt filename=blubber.yaml \
  --opt variant=test

If you'd like to build directly with docker build (or other toolchains that invoke it like docker-compose), specify a syntax directive at the top of your Blubber configuration like so.

# syntax=docker-registry.wikimedia.org/wikimedia/blubber-buildkit:v0.11.1
version: v4
variants:
  my-variant:
  [...]

And invoke docker build --target my-variant -f blubber.yaml .. Note that Docker must have BuildKit enabled as the default builder. You can also use docker buildx which always uses BuildKit.

Docker's build-time arguments are also supported, including those used to provide proxies to build processes.

buildctl build --frontend gateway.v0 \
  --opt source=docker-registry.wikimedia.org/wikimedia/blubber-buildkit:v0.11.1 \
  --opt build-arg:http_proxy=http://proxy.example \
  --opt variant=pulls-in-stuff-from-the-internet
  ...

Additional options for the Buildkit frontend

The following options can be passed via command line (via --opt) to configure the build process:

• run-variant: bool. Instructs Blubber to run the target variant's entrypoint (if any) as part of the BuildKit image build process
• entrypoint-args: JSON array. List of additional arguments for the entrypoint

Example usage:

buildctl build --frontend gateway.v0 \
  --opt source=docker-registry.wikimedia.org/wikimedia/blubber-buildkit:v0.11.1 \
  --local context=. \
  --local dockerfile=. \
  --opt filename=blubber.yaml \
  --opt variant=test \
  --opt run-variant=true
  --opt entrypoint-args='["extraParam1", "extraParam2"]'
  ...