task

package module
v0.11.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 11, 2023 License: Apache-2.0 Imports: 25 Imported by: 0

README

oci-build task

A Concourse task for building OCI images. Currently uses buildkit for building.

A stretch goal of this is to support running without privileged: true, though it currently still requires it.

usage

The task implementation is available as an image on Docker Hub at concourse/oci-build-task. (This image is built from Dockerfile using the oci-build task itself.)

This task implementation started as a spike to explore patterns around reusable tasks to hopefully lead to a proper RFC. Until that RFC is written and implemented, configuration is still done by way of providing your own task config as follows:

image_resource

First, your task needs to point to the oci-build-task image:

image_resource:
  type: registry-image
  source:
    repository: concourse/oci-build-task
params

Next, any of the following optional parameters may be specified:

(As a convention in the list below, all task parameters are specified with a leading $, in order to remind their environment variable nature, just like shell variables that one would use with the $VAR syntax. When specifying those in the params: YAML dictionary of a task definition though, the leading $ is irrelevant, as readers will notice in the examples below.)

  • $CONTEXT (default .): the path to the directory to provide as the context for the build.

  • $DOCKERFILE (default $CONTEXT/Dockerfile): the path to the Dockerfile to build.

  • $BUILDKIT_SSH your ssh key location that is mounted in your Dockerfile. This is generally used for pulling dependencies from private repositories.

    For Example. In your Dockerfile, you can mount a key as

    RUN --mount=type=ssh,id=github_ssh_key pip install -U -r ./hats/requirements-test.txt

    Then in your Concourse YAML configuration:

    params:
  BUILDKIT_SSH: github_ssh_key=<PATH-TO-YOUR-KEY>

    Read more about ssh mount here.

  • $BUILD_ARG_*: params prefixed with BUILD_ARG_ will be provided as build args. For example BUILD_ARG_foo=bar, will set the foo build arg as bar.

  • $BUILD_ARGS_FILE (default empty): path to a file containing build args in the form foo=bar, one per line. Empty lines are skipped.

    Example file contents:

    EMAIL=me@yopmail.com
HOW_MANY_THINGS=1
DO_THING=false

  • $BUILDKIT_SECRET_*: files with extra secrets which are made available via --mount=type=secret,id=.... See New Docker Build secret information for more information on build secrets.

    For example, running with BUILDKIT_SECRET_config=my-repo/config will allow you to do the following...

    RUN --mount=type=secret,id=config cat /run/secrets/config

  • $BUILDKIT_SECRETTEXT_*: literal text of extra secrets to be made available via the same mechanism described for $BUILDKIT_SECRET_* above. The difference is that this is easier to use with credential managers:

    BUILDKIT_SECRETTEXT_mysecret=(( mysecret )) puts the content that (( mysecret )) expands to in /run/secrets/mysecret.

  • $IMAGE_ARG_*: params prefixed with IMAGE_ARG_* point to image tarballs (i.e. docker save format) to preload so that they do not have to be fetched during the build. An image reference will be provided as the given build arg name. For example, IMAGE_ARG_base_image=ubuntu/image.tar will set base_image to a local image reference for using ubuntu/image.tar.

  • $IMAGE_PLATFORM: Specify the target platform to build the image for. For example IMAGE_PLATFORM=linux/arm64 will build the image for the Linux OS and arm64 architecture. By default, images will be built for the current worker's platform that the task is running on.

  • $LABEL_*: params prefixed with LABEL_ will be set as image labels. For example LABEL_foo=bar, will set the foo label to bar.

  • $LABELS_FILE (default empty): path to a file containing labels in the form foo=bar, one per line. Empty lines are skipped.

  • $TARGET (default empty): a target build stage to build, as named with the FROM … AS <NAME> syntax in your Dockerfile.

  • $TARGET_FILE (default empty): path to a file containing the name of the target build stage to build.

  • $ADDITIONAL_TARGETS (default empty): a comma-separated (,) list of additional target build stages to build.

  • $REGISTRY_MIRRORS (default empty): registry mirrors to use for docker.io.

  • $UNPACK_ROOTFS (default false): unpack the image as Concourse's image format (rootfs/, metadata.json) for use with the image task step option.

  • $OUTPUT_OCI (default false): outputs an OCI compliant image, allowing for multi-arch image builds when setting IMAGE_PLATFORM to [multiple platforms] (https://docs.docker.com/desktop/extensions-sdk/extensions/multi-arch/). The image output format will be a directory when this flag is set to true.

  • $BUILDKIT_ADD_HOSTS (default empty): extra host definitions for buildkit to properly resolve custom hostnames. The value is as comma-separated (,) list of key-value pairs (using syntax hostname=ip-address), each defining an IP address for resolving some custom hostname.

Note: this is the main pain point with reusable tasks - env vars are kind of an awkward way to configure a task. Once the RFC lands these will turn into a JSON structure similar to configuring params on a resource, and task params will become env instead.

inputs

There are no required inputs - your task should just list each artifact it needs as an input. Typically this is in close correlation with $CONTEXT:

params:
  CONTEXT: my-image

inputs:
- name: my-image

Should your build be dependent on multiple inputs, you may want to leave $CONTEXT as its default (.) and set an explicit path to the $DOCKERFILE:

params:
  DOCKERFILE: my-repo/Dockerfile

inputs:
- name: my-repo
- name: some-dependency

It might also make sense to place one input under another, like so:

params:
  CONTEXT: my-repo

inputs:
- name: my-repo
- name: some-dependency
  path: my-repo/some-dependency

Or, to fully rely on the default behavior and use path to wire up the context accordingly, you could set your primary context as path: . and set up any additional inputs underneath:

inputs:
- name: my-repo
  path: .
- name: some-dependency
outputs

A single output named image may be configured:

outputs:
- name: image

Use output_mapping to map this output to a different name in your build plan. This approach should be used if you're building multiple images in parallel so that they can have distinct names.

The output will contain the following files:

  • image.tar: the OCI image tarball. This tarball can be uploaded to a registry using the Registry Image resource.

  • digest: the digest of the OCI config. This file can be used to tag the image after it has been loaded with docker load, like so:

    docker load -i image/image.tar
docker tag $(cat image/digest) my-name

If $UNPACK_ROOTFS is configured, the following additional entries will be created:

  • rootfs/*: the unpacked contents of the image's filesystem.

  • metadata.json: a JSON file containing the image's env and user configuration.

This is a Concourse-specific format to support using the newly built image for a subsequent task by pointing the task step's image option to the output, like so:

plan:
- task: build-image
  params:
    UNPACK_ROOTFS: true
  output_mapping: {image: my-built-image}
- task: use-image
  image: my-built-image

(The output_mapping here is just for clarity; alternatively you could just set image: image.)

Note: at some point Concourse will likely standardize on OCI instead.

caches

Caching can be enabled by caching the cache path on the task:

caches:
- path: cache
run

Your task should run the build executable:

run:
  path: build

migrating from the docker-image resource

The docker-image resource was previously used for building and pushing a Docker image to a registry in one fell swoop.

The oci-build task, in contrast, only supports building images - it does not support pushing or even tagging the image. It can be used to build an image and use it for a subsequent task image without pushing it to a registry, by configuring $UNPACK_ROOTFS.

In order to push the newly built image, you can use a resource like the registry-image resource like so:

resources:
- name: my-image-src
  type: git
  source:
    uri: https://github.com/...

- name: my-image
  type: registry-image
  source:
    repository: my-user/my-repo

jobs:
- name: build-and-push
  plan:
  # fetch repository source (containing Dockerfile)
  - get: my-image-src

  # build using `oci-build` task
  #
  # note: this task config could be pushed into `my-image-src` and loaded using
  # `file:` instead
  - task: build
    privileged: true
    config:
      platform: linux

      image_resource:
        type: registry-image
        source:
          repository: concourse/oci-build-task

      inputs:
      - name: my-image-src
        path: .

      outputs:
      - name: image

      run:
        path: build

  # push using `registry-image` resource
  - put: my-image
    params: {image: image/image.tar}

differences from builder task

The builder task was a stepping stone that led to the oci-build task. It is now deprecated. The transition should be relatively smooth, with the following differences:

  • The oci-build task does not support configuring $REPOSITORY or $TAG.
    • for running the image with docker, a digest file is provided which can be tagged with docker tag
    • for pushing the image, the repository and tag are configured in the registry-image resource
  • The oci-build task has a more efficient caching implementation. By using buildkit directly we can make use of its local cache exporter/importer, which doesn't require a separate translation step for saving into the task cache.
  • This task is written in Go instead of Bash, and has tests!

example

This repo contains an example.yml, which builds the image for the task itself:

fly -t dev execute -c example.yml -o image=. -p
docker load -i image.tar

That -p at the end is not a typo; it runs the task with elevated privileges.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ServeRegistry 

func ServeRegistry(reg LocalRegistry) (string, error)

func StoreSecret added in v0.11.0 

func StoreSecret(req *Request, name, value string) error

Q: Audit name to not include "/"?

Types

type Buildkitd 

type Buildkitd struct {
	Addr string
	// contains filtered or unexported fields
}

func SpawnBuildkitd 

func SpawnBuildkitd(req Request, opts *BuildkitdOpts) (*Buildkitd, error)

func (*Buildkitd) Cleanup 

func (buildkitd *Buildkitd) Cleanup() error

type BuildkitdConfig 

type BuildkitdConfig struct {
	Registries map[string]RegistryConfig `toml:"registry"`
}

type BuildkitdOpts 

type BuildkitdOpts struct {
	RootDir    string
	ConfigPath string
}

BuildkitdOpts to provide to Buildkitd

type Config 

type Config struct {
	Debug bool `json:"debug" envconfig:"optional"`

	ContextDir     string `json:"context"              envconfig:"CONTEXT,optional"`
	DockerfilePath string `json:"dockerfile,omitempty" envconfig:"DOCKERFILE,optional"`
	BuildkitSSH    string `json:"buildkit_ssh"         envconfig:"optional"`

	Target            string   `json:"target"      envconfig:"optional"`
	TargetFile        string   `json:"target_file" envconfig:"optional"`
	AdditionalTargets []string `json:"additional_targets" envconfig:"ADDITIONAL_TARGETS,optional"`

	BuildArgs     []string `json:"build_args"      envconfig:"optional"`
	BuildArgsFile string   `json:"build_args_file" envconfig:"optional"`

	RegistryMirrors []string `json:"registry_mirrors" envconfig:"REGISTRY_MIRRORS,optional"`

	Labels     []string `json:"labels"      envconfig:"optional"`
	LabelsFile string   `json:"labels_file" envconfig:"optional"`

	BuildkitSecrets map[string]string `json:"buildkit_secrets" envconfig:"optional"`

	// Unpack the OCI image into Concourse's rootfs/ + metadata.json image scheme.
	//
	// Theoretically this would go away if/when we standardize on OCI.
	UnpackRootfs bool `json:"unpack_rootfs" envconfig:"optional"`

	OutputOCI bool `json:"output_oci" envconfig:"optional"`

	// Images to pre-load in order to avoid fetching at build time. Mapping from
	// build arg name to OCI image tarball path.
	//
	// Each image will be pre-loaded and a build arg will be set to a value
	// appropriate for setting in 'FROM ...'.
	ImageArgs []string `json:"image_args" envconfig:"optional"`

	AddHosts string `json:"add_hosts" envconfig:"BUILDKIT_ADD_HOSTS,optional"`

	ImagePlatform string `json:"image_platform" envconfig:"optional"`
}

Config contains the configuration for the task.

In the future, when Concourse supports a 'reusable task' interface, this will be provided as a JSON request on `stdin`.

For now, and for backwards-compatibility, we will also support taking values from task params (i.e. env), hence the use of `envconfig:`.

type ImageArg added in v0.9.1 

type ImageArg struct {
	Image   v1.Image
	ArgName string
}

type ImageMetadata 

type ImageMetadata struct {
	Env  []string `json:"env"`
	User string   `json:"user"`
}

ImageMetadata is the schema written to manifest.json when producing the legacy Concourse image format (rootfs/..., metadata.json).

type LocalRegistry 

type LocalRegistry map[string]ImageArg

func LoadRegistry 

func LoadRegistry(imagePaths map[string]string) (LocalRegistry, error)

func (LocalRegistry) BuildArgs 

func (registry LocalRegistry) BuildArgs(port string) []string

func (LocalRegistry) GetBlob 

func (registry LocalRegistry) GetBlob(w http.ResponseWriter, r *http.Request, p httprouter.Params)

func (LocalRegistry) GetManifest 

func (registry LocalRegistry) GetManifest(w http.ResponseWriter, r *http.Request, p httprouter.Params)

type RegistryConfig 

type RegistryConfig struct {
	Mirrors      []string     `toml:"mirrors"`
	PlainHTTP    *bool        `toml:"http"`
	Insecure     *bool        `toml:"insecure"`
	RootCAs      []string     `toml:"ca"`
	KeyPairs     []TLSKeyPair `toml:"keypair"`
	TLSConfigDir []string     `toml:"tlsconfigdir"`
}

type Request 

type Request struct {
	ResponsePath string `json:"response_path"`
	Config       Config `json:"config"`
}

Request is the request payload sent from Concourse to execute the task.

This is currently not really exercised by Concourse; it's a mock-up of what a future 'reusable tasks' design may look like.

type Response 

type Response struct {
	Outputs []string `json:"outputs"`
}

Response is sent back to Concourse by writing this structure to the `response_path` specified in the request.

This is also a mock-up. Right now it communicates the available outputs, which may be useful to assist pipeline authors in knowing what artifacts are available after a task excutes.

In the future, pipeline authors may list which outputs they would like to propagate to the rest of the build plan, by specifying `outputs` or `output_mapping` like so: 

task: build
outputs: [image]

task: build
output_mapping: {image: my-image}

Outputs may also be 'cached', meaning their previous value will be present for subsequent runs of the task: 

task: build
outputs: [image]
caches: [cache]

func Build 

func Build(buildkitd *Buildkitd, outputsDir string, req Request) (Response, error)

type TLSConfig 

type TLSConfig struct {
	Cert string `toml:"cert"`
	Key  string `toml:"key"`
	CA   string `toml:"ca"`
}

type TLSKeyPair 

type TLSKeyPair struct {
	Key         string `toml:"key"`
	Certificate string `toml:"cert"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
build
task

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.