tarball

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 22, 2020 License: Apache-2.0 Imports: 21 Imported by: 294

README

tarball

GoDoc

This package produces tarballs that can consumed via docker load. Note that this is a different format from the legacy tarballs that are produced by docker save, but this package is still able to read the legacy tarballs produced by docker save.

Usage

package main

import (
	"os"

	"github.com/google/go-containerregistry/pkg/name"
	"github.com/google/go-containerregistry/pkg/v1/tarball"
)

func main() {
	// Read a tarball from os.Args[1] that contains ubuntu.
	tag, err := name.NewTag("ubuntu")
	if err != nil {
		panic(err)
	}
	img, err := tarball.ImageFromPath(os.Args[1], &tag)
	if err != nil {
		panic(err)
	}

	// Write that tarball to os.Args[2] with a different tag.
	newTag, err := name.NewTag("ubuntu:newest")
	if err != nil {
		panic(err)
	}
	f, err := os.Create(os.Args[2])
	if err != nil {
		panic(err)
	}
	defer f.Close()

	if err := tarball.Write(newTag, img, f); err != nil {
		panic(err)
	}
}

Structure

Let's look at what happens when we write out a tarball:

ubuntu:latest
$ crane pull ubuntu ubuntu.tar && mkdir ubuntu && tar xf ubuntu.tar -C ubuntu && rm ubuntu.tar
$ tree ubuntu/
ubuntu/
├── 423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954.tar.gz
├── b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7.tar.gz
├── de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d.tar.gz
├── f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b.tar.gz
├── manifest.json
└── sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c

0 directories, 6 files

There are a couple interesting files here.

manifest.json is the entrypoint: a list of tarball.Descriptors that describe the images contained in this tarball.

For each image, this has the RepoTags (how it was pulled), a Config file that points to the image's config file, a list of Layers, and (optionally) LayerSources.

$ jq < ubuntu/manifest.json
[
  {
    "Config": "sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c",
    "RepoTags": [
      "ubuntu"
    ],
    "Layers": [
      "423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954.tar.gz",
      "de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d.tar.gz",
      "f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b.tar.gz",
      "b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7.tar.gz"
    ]
  }
]

The config file and layers are exactly what you would expect, and match the registry representations of the same artifacts. You'll notice that the manifest.json contains similar information as the registry manifest, but isn't quite the same:

$ crane manifest ubuntu@sha256:0925d086715714114c1988f7c947db94064fd385e171a63c07730f1fa014e6f9
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 3408,
      "digest": "sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 26692096,
         "digest": "sha256:423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 35365,
         "digest": "sha256:de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 852,
         "digest": "sha256:f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 163,
         "digest": "sha256:b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7"
      }
   ]
}

This makes it difficult to maintain image digests when roundtripping images through the tarball format, so it's not a great format if you care about provenance.

The ubuntu example didn't have any LayerSources -- let's look at another image that does.

hello-world:nanoserver
$ crane pull hello-world:nanoserver@sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b nanoserver.tar
$ mkdir nanoserver && tar xf nanoserver.tar -C nanoserver && rm nanoserver.tar
$ tree nanoserver/
nanoserver/
├── 10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0.tar.gz
├── a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053.tar.gz
├── be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a.tar.gz
├── manifest.json
└── sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6

0 directories, 5 files

$ jq < nanoserver/manifest.json
[
  {
    "Config": "sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6",
    "RepoTags": [
      "index.docker.io/library/hello-world:i-was-a-digest"
    ],
    "Layers": [
      "a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053.tar.gz",
      "be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a.tar.gz",
      "10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0.tar.gz"
    ],
    "LayerSources": {
      "sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e": {
        "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
        "size": 101145811,
        "digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
        "urls": [
          "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
        ]
      }
    }
  }
]

A couple things to note about this manifest.json versus the other:

  • The RepoTags field is a bit weird here. hello-world is a multi-platform image, so We had to pull this image by digest, since we're (I'm) on amd64/linux and wanted to grab a windows image. Since the tarball format expects a tag under RepoTags, and we didn't pull by tag, we replace the digest with a sentinel i-was-a-digest "tag" to appease docker.
  • The LayerSources has enough information to reconstruct the foreign layers pointer when pushing/pulling from the registry. For legal reasons, microsoft doesn't want anyone but them to serve windows base images, so the mediaType here indicates a "foreign" or "non-distributable" layer with an URL for where you can download it from microsoft (see the OCI image-spec).

We can look at what's in the registry to explain both of these things:

$ crane manifest hello-world:nanoserver | jq .
{
  "manifests": [
    {
      "digest": "sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "windows",
        "os.version": "10.0.17763.1040"
      },
      "size": 1124
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}


# Note the media type and "urls" field.
$ crane manifest hello-world:nanoserver@sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1721,
    "digest": "sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
      "size": 101145811,
      "digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
      "urls": [
        "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
      ]
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1669,
      "digest": "sha256:be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 949,
      "digest": "sha256:10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0"
    }
  ]
}

The LayerSources map is keyed by the diffid. Note that sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e matches the first layer in the config file:

$ jq '.[0].LayerSources' < nanoserver/manifest.json
{
  "sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e": {
    "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
    "size": 101145811,
    "digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
    "urls": [
      "https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
    ]
  }
}

$ jq < nanoserver/sha256\:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6 | jq .rootfs
{
  "type": "layers",
  "diff_ids": [
    "sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e",
    "sha256:601cf7d78c62e4b4d32a7bbf96a17606a9cea5bd9d22ffa6f34aa431d056b0e8",
    "sha256:a1e1a3bf6529adcce4d91dce2cad86c2604a66b507ccbc4d2239f3da0ec5aab9"
  ]
}

Documentation

Overview

Package tarball provides facilities for reading/writing v1.Images from/to a tarball on-disk.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

func CalculateSize added in v0.2.0 

func CalculateSize(refToImage map[name.Reference]v1.Image) (size int64, err error)

CalculateSize calculates the expected complete size of the output tar file

func Image 

func Image(opener Opener, tag *name.Tag) (v1.Image, error)

Image exposes an image from the tarball at the provided path.

func ImageFromPath 

func ImageFromPath(path string, tag *name.Tag) (v1.Image, error)

ImageFromPath returns a v1.Image from a tarball located on path.

func LayerFromFile 

func LayerFromFile(path string, opts ...LayerOption) (v1.Layer, error)

LayerFromFile returns a v1.Layer given a tarball

func LayerFromOpener 

func LayerFromOpener(opener Opener, opts ...LayerOption) (v1.Layer, error)

LayerFromOpener returns a v1.Layer given an Opener function. The Opener may return either an uncompressed tarball (common), or a compressed tarball (uncommon).

When using this in conjunction with something like remote.Write the uncompressed path may end up gzipping things multiple times:

  1. Compute the layer SHA256
  2. Upload the compressed layer.

Since gzip can be expensive, we support an option to memoize the compression that can be passed here: tarball.WithCompressedCaching

func LayerFromReader 

func LayerFromReader(reader io.Reader, opts ...LayerOption) (v1.Layer, error)

LayerFromReader returns a v1.Layer given a io.Reader.

func MultiRefWrite 

func MultiRefWrite(refToImage map[name.Reference]v1.Image, w io.Writer, opts ...WriteOption) error

MultiRefWrite writes the contents of each image to the provided reader, in the compressed format. The contents are written in the following format: One manifest.json file at the top level containing information about several images. One file for each layer, named after the layer's SHA. One file for the config blob, named after its SHA.

func MultiRefWriteToFile 

func MultiRefWriteToFile(p string, refToImage map[name.Reference]v1.Image, opts ...WriteOption) error

MultiRefWriteToFile writes in the compressed format to a tarball, on disk. This is just syntactic sugar wrapping tarball.MultiRefWrite with a new file.

func MultiWrite 

func MultiWrite(tagToImage map[name.Tag]v1.Image, w io.Writer, opts ...WriteOption) error

MultiWrite writes the contents of each image to the provided reader, in the compressed format. The contents are written in the following format: One manifest.json file at the top level containing information about several images. One file for each layer, named after the layer's SHA. One file for the config blob, named after its SHA.

func MultiWriteToFile 

func MultiWriteToFile(p string, tagToImage map[name.Tag]v1.Image, opts ...WriteOption) error

MultiWriteToFile writes in the compressed format to a tarball, on disk. This is just syntactic sugar wrapping tarball.MultiWrite with a new file.

func WithCompressedCaching added in v0.3.0 

func WithCompressedCaching(l *layer)

WithCompressedCaching is a functional option that overrides the logic for accessing the compressed bytes to memoize the result and avoid expensive repeated gzips.

func WithEstargz added in v0.3.0 

func WithEstargz(l *layer)

WithEstargz is a functional option that explicitly enables estargz support.

func Write 

func Write(ref name.Reference, img v1.Image, w io.Writer, opts ...WriteOption) error

Write is a wrapper to write a single image and tag to a tarball.

func WriteToFile 

func WriteToFile(p string, ref name.Reference, img v1.Image, opts ...WriteOption) error

WriteToFile writes in the compressed format to a tarball, on disk. This is just syntactic sugar wrapping tarball.Write with a new file.

Types

type Descriptor 

type Descriptor struct {
	Config   string
	RepoTags []string
	Layers   []string

	// Tracks foreign layer info. Key is DiffID.
	LayerSources map[v1.Hash]v1.Descriptor `json:",omitempty"`
}

Descriptor stores the manifest data for a single image inside a `docker save` tarball.

type LayerOption 

type LayerOption func(*layer)

LayerOption applies options to layer

func WithCompressionLevel 

func WithCompressionLevel(level int) LayerOption

WithCompressionLevel is a functional option for overriding the default compression level used for compressing uncompressed tarballs.

func WithEstargzOptions added in v0.3.0 

func WithEstargzOptions(opts ...estargz.Option) LayerOption

WithEstargzOptions is a functional option that allow the caller to pass through estargz.Options to the underlying compression layer. This is only meaningful when estargz is enabled.

type Manifest 

type Manifest []Descriptor

Manifest represents the manifests of all images as the `manifest.json` file in a `docker save` tarball.

func ComputeManifest 

func ComputeManifest(refToImage map[name.Reference]v1.Image) (Manifest, error)

ComputeManifest get the manifest.json that will be written to the tarball for multiple references

type Opener 

type Opener func() (io.ReadCloser, error)

Opener is a thunk for opening a tar file.

type WriteOption 

type WriteOption func(*writeOptions) error

WriteOption a function option to pass to Write()

func WithProgress 

func WithProgress(updates chan<- v1.Update) WriteOption

WithProgress create a WriteOption for passing to Write() that enables a channel to receive updates as they are downloaded and written to disk.

Example 
package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"os"

	"github.com/google/go-containerregistry/pkg/name"
	v1 "github.com/google/go-containerregistry/pkg/v1"
	"github.com/google/go-containerregistry/pkg/v1/remote"
	"github.com/google/go-containerregistry/pkg/v1/tarball"
)

func main() {
	/* calculations for this test:
	The image we are using is docker.io/library/alpine:3.10
	its size on disk is 2800640
	The filesizes inside are:
	-rw-r--r--  0 0      0        1509 Jan  1  1970 sha256:be4e4bea2c2e15b403bb321562e78ea84b501fb41497472e91ecb41504e8a27c
	-rw-r--r--  0 0      0     2795580 Jan  1  1970 21c83c5242199776c232920ddb58cfa2a46b17e42ed831ca9001c8dbc532d22d.tar.gz
	-rw-r--r--  0 0      0         216 Jan  1  1970 manifest.json
	when rounding each to a 512-byte block, plus the header, we get:
	1509    ->    1536 + 512 = 2048
	2795580 -> 2796032 + 512 = 2796544
	216     ->     512 + 512 = 1024
	add in 2 blocks of all 0x00 to indicate end of archive
	                         = 1024
	                        -------
	Total:                  2800640
	*/
	// buffered channel to make the example test easier
	c := make(chan v1.Update, 200)
	// Make a tempfile for tarball writes.
	fp, err := ioutil.TempFile("", "")
	if err != nil {
		fmt.Printf("error creating temp file: %v\n", err)
		return
	}
	defer fp.Close()
	defer os.Remove(fp.Name())

	tag, err := name.NewDigest("docker.io/library/alpine@sha256:f0e9534a598e501320957059cb2a23774b4d4072e37c7b2cf7e95b241f019e35", name.StrictValidation)
	if err != nil {
		fmt.Printf("error creating test tag: %v\n", err)
		return
	}
	desc, err := remote.Get(tag)
	if err != nil {
		fmt.Printf("error getting manifest: %v", err)
		return
	}
	img, err := desc.Image()
	if err != nil {
		fmt.Printf("error image: %v", err)
		return
	}
	go func() {
		_ = tarball.WriteToFile(fp.Name(), tag, img, tarball.WithProgress(c))
	}()
	for update := range c {
		switch {
		case update.Error != nil && update.Error == io.EOF:
			fmt.Fprintf(os.Stderr, "receive error message: %v\n", err)
			fmt.Printf("%d/%d", update.Complete, update.Total)

			return
		case update.Error != nil:
			fmt.Printf("error writing tarball: %v\n", update.Error)
			return
		default:
			fmt.Fprintf(os.Stderr, "receive update: %#v\n", update)
		}
	}
}

Output:

2800640/2800640

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.