tarball

README

tarball

This package produces tarballs that can consumed via docker load. Note that this is a different format from the legacy tarballs that are produced by docker save, but this package is still able to read the legacy tarballs produced by docker save.

Usage

package main

import (
	"os"

	"github.com/google/go-containerregistry/pkg/name"
	"github.com/google/go-containerregistry/pkg/v1/tarball"
)

func main() {
	// Read a tarball from os.Args[1] that contains ubuntu.
	tag, err := name.NewTag("ubuntu")
	if err != nil {
		panic(err)
	}
	img, err := tarball.ImageFromPath(os.Args[1], &tag)
	if err != nil {
		panic(err)
	}

	// Write that tarball to os.Args[2] with a different tag.
	newTag, err := name.NewTag("ubuntu:newest")
	if err != nil {
		panic(err)
	}
	f, err := os.Create(os.Args[2])
	if err != nil {
		panic(err)
	}
	defer f.Close()

	if err := tarball.Write(newTag, img, f); err != nil {
		panic(err)
	}
}

Structure

Let's look at what happens when we write out a tarball:

ubuntu:latest

$ crane pull ubuntu ubuntu.tar && mkdir ubuntu && tar xf ubuntu.tar -C ubuntu && rm ubuntu.tar
$ tree ubuntu/
ubuntu/
├── 423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954.tar.gz
├── b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7.tar.gz
├── de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d.tar.gz
├── f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b.tar.gz
├── manifest.json
└── sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c

0 directories, 6 files

There are a couple interesting files here.

manifest.json is the entrypoint: a list of tarball.Descriptors that describe the images containd in this tarball.

For each image, this has the RepoTags (how it was pulled), a Config file that points to the image's config file, a list of Layers, and (optionally) LayerSources.

$ jq < ubuntu/manifest.json
[
  {
    "Config": "sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c",
    "RepoTags": [
      "ubuntu"
    ],
    "Layers": [
      "423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954.tar.gz",
      "de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d.tar.gz",
      "f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b.tar.gz",
      "b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7.tar.gz"
    ]
  }
]

The config file and layers are exactly what you would expect, and match the registry representations of the same artifacts. You'll notice that the manifest.json contains similar information as the registry manifest, but isn't quite the same:

$ crane manifest ubuntu@sha256:0925d086715714114c1988f7c947db94064fd385e171a63c07730f1fa014e6f9
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 3408,
      "digest": "sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 26692096,
         "digest": "sha256:423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 35365,
         "digest": "sha256:de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 852,
         "digest": "sha256:f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 163,
         "digest": "sha256:b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7"
      }
   ]
}

This makes it difficult to maintain image digests when roundtripping images through the tarball format, so it's not a great format if you care about provenance.

The ubuntu example didn't have any LayerSources -- let's look at another image that does.

hello-world:nanoserver

$ crane pull hello-world:nanoserver@sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b nanoserver.tar
$ mkdir nanoserver && tar xf nanoserver.tar -C nanoserver && rm nanoserver.tar
$ tree nanoserver/
nanoserver/
├── 10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0.tar.gz
├── a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053.tar.gz
├── be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a.tar.gz
├── manifest.json
└── sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6

0 directories, 5 files

$ jq < nanoserver/manifest.json
[
  {
    "Config": "sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6",
    "RepoTags": [
      "index.docker.io/library/hello-world:i-was-a-digest"
    ],
    "Layers": [
      "a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053.tar.gz",
      "be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a.tar.gz",
      "10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0.tar.gz"
    ],
    "LayerSources": {
      "sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e": {
        "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
        "size": 101145811,
        "digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
        "urls": [
          "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
        ]
      }
    }
  }
]

A couple things to note about this manifest.json versus the other:

• The RepoTags field is a bit weird here. hello-world is a multi-platform image, so We had to pull this image by digest, since we're (I'm) on amd64/linux and wanted to grab a windows image. Since the tarball format expects a tag under RepoTags, and we didn't pull by tag, we replace the digest with a sentinel i-was-a-digest "tag" to appease docker.
• The LayerSources has enough information to reconstruct the foreign layers pointer when pushing/pulling from the registry. For legal reasons, microsoft doesn't want anyone but them to serve windows base images, so the mediaType here indicates a "foreign" or "non-distributable" layer with an URL for where you can download it from microsoft (see the OCI image-spec).

We can look at what's in the registry to explain both of these things:

$ crane manifest hello-world:nanoserver | jq .
{
  "manifests": [
    {
      "digest": "sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "windows",
        "os.version": "10.0.17763.1040"
      },
      "size": 1124
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}


# Note the media type and "urls" field.
$ crane manifest hello-world:nanoserver@sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1721,
    "digest": "sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
      "size": 101145811,
      "digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
      "urls": [
        "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
      ]
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1669,
      "digest": "sha256:be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 949,
      "digest": "sha256:10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0"
    }
  ]
}

The LayerSources map is keyed by the diffid. Note that sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e matches the first layer in the config file:

$ jq '.[0].LayerSources' < nanoserver/manifest.json
{
  "sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e": {
    "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
    "size": 101145811,
    "digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
    "urls": [
      "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
    ]
  }
}

$ jq < nanoserver/sha256\:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6 | jq .rootfs
{
  "type": "layers",
  "diff_ids": [
    "sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e",
    "sha256:601cf7d78c62e4b4d32a7bbf96a17606a9cea5bd9d22ffa6f34aa431d056b0e8",
    "sha256:a1e1a3bf6529adcce4d91dce2cad86c2604a66b507ccbc4d2239f3da0ec5aab9"
  ]
}

Documentation

Overview

Package tarball provides facilities for reading/writing v1.Images from/to a tarball on-disk.

Index

• func CalculateTarFileSize(in int64) (out int64)
• func Image(opener Opener, tag *name.Tag) (v1.Image, error)
• func ImageFromPath(path string, tag *name.Tag) (v1.Image, error)
• func LayerFromFile(path string, opts ...LayerOption) (v1.Layer, error)
• func LayerFromOpener(opener Opener, opts ...LayerOption) (v1.Layer, error)
• func LayerFromReader(reader io.Reader, opts ...LayerOption) (v1.Layer, error)
• func MultiRefWrite(refToImage map[name.Reference]v1.Image, w io.Writer, opts ...WriteOption) error
• func MultiRefWriteToFile(p string, refToImage map[name.Reference]v1.Image, opts ...WriteOption) error
• func MultiWrite(tagToImage map[name.Tag]v1.Image, w io.Writer, opts ...WriteOption) error
• func MultiWriteToFile(p string, tagToImage map[name.Tag]v1.Image, opts ...WriteOption) error
• func Write(ref name.Reference, img v1.Image, w io.Writer, opts ...WriteOption) error
• func WriteToFile(p string, ref name.Reference, img v1.Image, opts ...WriteOption) error
• type Descriptor
• type LayerOption
  • func WithCompressionLevel(level int) LayerOption
• type Manifest
  • func ComputeManifest(refToImage map[name.Reference]v1.Image) (Manifest, error)
• type Opener
• type WriteOption
  • func WithProgress(updates chan<- v1.Update) WriteOption

Examples

• WithProgress

Functions

func CalculateTarFileSize

func CalculateTarFileSize(in int64) (out int64)

CalculateTarFileSize calculate the size a file will take up in a tar archive, given the input data. Provided by rounding up to nearest whole block (512) and adding header 512

func Image

func Image(opener Opener, tag *name.Tag) (v1.Image, error)

Image exposes an image from the tarball at the provided path.

func ImageFromPath

func ImageFromPath(path string, tag *name.Tag) (v1.Image, error)

ImageFromPath returns a v1.Image from a tarball located on path.

func LayerFromFile

func LayerFromFile(path string, opts ...LayerOption) (v1.Layer, error)

LayerFromFile returns a v1.Layer given a tarball

func LayerFromOpener

func LayerFromOpener(opener Opener, opts ...LayerOption) (v1.Layer, error)

LayerFromOpener returns a v1.Layer given an Opener function

func LayerFromReader

func LayerFromReader(reader io.Reader, opts ...LayerOption) (v1.Layer, error)

LayerFromReader returns a v1.Layer given a io.Reader.

func MultiRefWrite

func MultiRefWrite(refToImage map[name.Reference]v1.Image, w io.Writer, opts ...WriteOption) error

MultiRefWrite writes the contents of each image to the provided reader, in the compressed format. The contents are written in the following format: One manifest.json file at the top level containing information about several images. One file for each layer, named after the layer's SHA. One file for the config blob, named after its SHA.

func MultiRefWriteToFile

func MultiRefWriteToFile(p string, refToImage map[name.Reference]v1.Image, opts ...WriteOption) error

MultiRefWriteToFile writes in the compressed format to a tarball, on disk. This is just syntactic sugar wrapping tarball.MultiRefWrite with a new file.

func MultiWrite

func MultiWrite(tagToImage map[name.Tag]v1.Image, w io.Writer, opts ...WriteOption) error

MultiWrite writes the contents of each image to the provided reader, in the compressed format. The contents are written in the following format: One manifest.json file at the top level containing information about several images. One file for each layer, named after the layer's SHA. One file for the config blob, named after its SHA.

func MultiWriteToFile

func MultiWriteToFile(p string, tagToImage map[name.Tag]v1.Image, opts ...WriteOption) error

MultiWriteToFile writes in the compressed format to a tarball, on disk. This is just syntactic sugar wrapping tarball.MultiWrite with a new file.

func Write

func Write(ref name.Reference, img v1.Image, w io.Writer, opts ...WriteOption) error

Write is a wrapper to write a single image and tag to a tarball.

func WriteToFile

func WriteToFile(p string, ref name.Reference, img v1.Image, opts ...WriteOption) error

WriteToFile writes in the compressed format to a tarball, on disk. This is just syntactic sugar wrapping tarball.Write with a new file.

Types

type Descriptor

type Descriptor struct {
	Config   string
	RepoTags []string
	Layers   []string

	// Tracks foreign layer info. Key is DiffID.
	LayerSources map[v1.Hash]v1.Descriptor `json:",omitempty"`
}

Descriptor stores the manifest data for a single image inside a `docker save` tarball.

type LayerOption

type LayerOption func(*layer)

LayerOption applies options to layer

func WithCompressionLevel

func WithCompressionLevel(level int) LayerOption

WithCompressionLevel sets the gzip compression. See `gzip.NewWriterLevel` for possible values.

type Manifest

type Manifest []Descriptor

Manifest represents the manifests of all images as the `manifest.json` file in a `docker save` tarball.

func ComputeManifest

func ComputeManifest(refToImage map[name.Reference]v1.Image) (Manifest, error)

ComputeManifest get the manifest.json that will be written to the tarball for multiple references

type Opener

type Opener func() (io.ReadCloser, error)

Opener is a thunk for opening a tar file.

type WriteOption

type WriteOption func(*writeOptions) error

WriteOption a function option to pass to Write()

func WithProgress

func WithProgress(updates chan<- v1.Update) WriteOption

WithProgress create a WriteOption for passing to Write() that enables a channel to receive updates as they are downloaded and written to disk.

Example

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"os"

	"github.com/google/go-containerregistry/pkg/name"
	v1 "github.com/google/go-containerregistry/pkg/v1"
	"github.com/google/go-containerregistry/pkg/v1/remote"
	"github.com/google/go-containerregistry/pkg/v1/tarball"
)

func main() {
	/* calculations for this test:
	The image we are using is docker.io/library/alpine:3.10
	its size on disk is 2800640
	The filesizes inside are:
	-rw-r--r--  0 0      0        1509 Jan  1  1970 sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c
	-rw-r--r--  0 0      0     2795580 Jan  1  1970 21c83c5242199776c232920ddb58cfa2a46b17e42ed831ca9001c8dbc532d22d.tar.gz
	-rw-r--r--  0 0      0         216 Jan  1  1970 manifest.json
	when rounding each to a 512-byte block, plus the header, we get:
	1509    ->    1536 + 512 = 2048
	2795580 -> 2796032 + 512 = 2796544
	216     ->     512 + 512 = 1024
	add in 2 blocks of all 0x00 to indicate end of archive
	                         = 1024
	                        -------
	Total:                  2800640
	*/
	// buffered channel to make the example test easier
	c := make(chan v1.Update, 200)
	// Make a tempfile for tarball writes.
	fp, err := ioutil.TempFile("", "")
	if err != nil {
		fmt.Printf("error creating temp file: %v\n", err)
		return
	}
	defer fp.Close()
	defer os.Remove(fp.Name())

	tag, err := name.NewDigest("docker.io/library/alpine@sha256:f0e9534a598e501320957059cb2a23774b4d4072e37c7b2cf7e95b241f019e35", name.StrictValidation)
	if err != nil {
		fmt.Printf("error creating test tag: %v\n", err)
		return
	}
	desc, err := remote.Get(tag)
	if err != nil {
		fmt.Printf("error getting manifest: %v", err)
		return
	}
	img, err := desc.Image()
	if err != nil {
		fmt.Printf("error image: %v", err)
		return
	}
	go func() {
		_ = tarball.WriteToFile(fp.Name(), tag, img, tarball.WithProgress(c))
	}()
	for update := range c {
		switch {
		case update.Error != nil && update.Error == io.EOF:
			fmt.Fprintf(os.Stderr, "receive error message: %v\n", err)
			fmt.Printf("%d/%d", update.Complete, update.Total)

			return
		case update.Error != nil:
			fmt.Printf("error writing tarball: %v\n", update.Error)
			return
		default:
			fmt.Fprintf(os.Stderr, "receive update: %#v\n", update)
		}
	}
}

Output:

2800640/2800640