minimal

Documentation

Index

• Constants
• Variables
• func GetType(t byte) (string, error)
• func WriteZstdChunkedManifest(dest io.Writer, outMetadata map[string]string, offset uint64, ...) error
• func ZstdWriterWithLevel(dest io.Writer, level int) (*zstd.Encoder, error)
• type FileMetadata
  • func NewFileMetadata(hdr *tar.Header) (FileMetadata, error)
• type TOC
• type TarSplitData
• type ZstdChunkedFooterData

Constants

const (
	ChunkTypeData  = ""
	ChunkTypeZeros = "zeros"
)

const (
	// The following types correspond to regular types of entries that can
	// appear in a tar archive.
	TypeReg     = "reg"
	TypeLink    = "hardlink"
	TypeChar    = "char"
	TypeBlock   = "block"
	TypeDir     = "dir"
	TypeFifo    = "fifo"
	TypeSymlink = "symlink"
	// TypeChunk is special; in zstd:chunked not only are files individually
	// compressed and indexable, there is a "rolling checksum" used to compute
	// "chunks" of individual file contents, that are also added to the TOC
	TypeChunk = "chunk"
)

const (
	// ManifestChecksumKey is a hexadecimal sha256 digest of the compressed manifest digest.
	ManifestChecksumKey = "io.github.containers.zstd-chunked.manifest-checksum"
	// ManifestInfoKey is an annotation that signals the start of the TOC (manifest)
	// contents which are embedded as a skippable zstd frame.  It has a format of
	// four decimal integers separated by `:` as follows:
	// <offset>:<length>:<uncompressed length>:<type>
	// The <type> is ManifestTypeCRFS which should have the value `1`.
	ManifestInfoKey = "io.github.containers.zstd-chunked.manifest-position"
	// TarSplitInfoKey is an annotation that signals the start of the "tar-split" metadata
	// contents which are embedded as a skippable zstd frame.  It has a format of
	// three decimal integers separated by `:` as follows:
	// <offset>:<length>:<uncompressed length>
	TarSplitInfoKey = "io.github.containers.zstd-chunked.tarsplit-position"

	// ManifestTypeCRFS is a manifest file compatible with the CRFS TOC file.
	ManifestTypeCRFS = 1

	// FooterSizeSupported is the footer size supported by this implementation.
	// Newer versions of the image format might increase this value, so reject
	// any version that is not supported.
	FooterSizeSupported = 64
)

Variables

var TarTypes = map[byte]string{
	tar.TypeReg:     TypeReg,
	tar.TypeLink:    TypeLink,
	tar.TypeChar:    TypeChar,
	tar.TypeBlock:   TypeBlock,
	tar.TypeDir:     TypeDir,
	tar.TypeFifo:    TypeFifo,
	tar.TypeSymlink: TypeSymlink,
}

var (
	ZstdChunkedFrameMagic = []byte{0x47, 0x4e, 0x55, 0x6c, 0x49, 0x6e, 0x55, 0x78}
)

Functions

func GetType

func GetType(t byte) (string, error)

func WriteZstdChunkedManifest

func WriteZstdChunkedManifest(dest io.Writer, outMetadata map[string]string, offset uint64, tarSplitData *TarSplitData, metadata []FileMetadata, level int) error

func ZstdWriterWithLevel

func ZstdWriterWithLevel(dest io.Writer, level int) (*zstd.Encoder, error)

Types

type FileMetadata

type FileMetadata struct {

	// The metadata below largely duplicates that in the tar headers.
	Type       string            `json:"type"`
	Name       string            `json:"name"`
	Linkname   string            `json:"linkName,omitempty"`
	Mode       int64             `json:"mode,omitempty"`
	Size       int64             `json:"size,omitempty"`
	UID        int               `json:"uid,omitempty"`
	GID        int               `json:"gid,omitempty"`
	ModTime    *time.Time        `json:"modtime,omitempty"`
	AccessTime *time.Time        `json:"accesstime,omitempty"`
	ChangeTime *time.Time        `json:"changetime,omitempty"`
	Devmajor   int64             `json:"devMajor,omitempty"`
	Devminor   int64             `json:"devMinor,omitempty"`
	Xattrs     map[string]string `json:"xattrs,omitempty"`
	// Digest is a hexadecimal sha256 checksum of the file contents; it
	// is empty for empty files
	Digest    string `json:"digest,omitempty"`
	Offset    int64  `json:"offset,omitempty"`
	EndOffset int64  `json:"endOffset,omitempty"`

	ChunkSize   int64  `json:"chunkSize,omitempty"`
	ChunkOffset int64  `json:"chunkOffset,omitempty"`
	ChunkDigest string `json:"chunkDigest,omitempty"`
	ChunkType   string `json:"chunkType,omitempty"`
}

FileMetadata is an entry in the TOC that includes both generic file metadata that duplicates what can found in the tar header (and should match), but also special/custom content (see below).

Regular files may optionally be represented as a sequence of “chunks”, which may be ChunkTypeData or ChunkTypeZeros (and ChunkTypeData boundaries are heuristically determined to increase chance of chunk matching / reuse similar to rsync). In that case, the regular file is represented as an initial TypeReg entry (with all metadata for the file as a whole) immediately followed by zero or more TypeChunk entries (containing only Type, Name and Chunk* fields); if there is at least one TypeChunk entry, the Chunk* fields are relevant in all of these entries, including the initial TypeReg one.

Note that the metadata here, when fetched by a zstd:chunked aware client, is used instead of that in the tar stream. The contents of the tar stream are not used in this scenario.

func NewFileMetadata

func NewFileMetadata(hdr *tar.Header) (FileMetadata, error)

NewFileMetadata creates a basic FileMetadata entry for hdr. The caller must set DigestOffset/EndOffset, and the Chunk* values, separately.

type TOC

type TOC struct {
	// Version is currently expected to be 1
	Version int `json:"version"`
	// Entries is the list of file metadata in this TOC.
	// The ordering in this array currently defaults to being the same
	// as that of the tar stream; however, this should not be relied on.
	Entries []FileMetadata `json:"entries"`
	// TarSplitDigest is the checksum of the "tar-split" data which
	// is included as a distinct skippable zstd frame before the TOC.
	TarSplitDigest digest.Digest `json:"tarSplitDigest,omitempty"`
}

TOC is short for Table of Contents and is used by the zstd:chunked file format to effectively add an overall index into the contents of a tarball; it also includes file metadata.

type TarSplitData

type TarSplitData struct {
	Data             []byte
	Digest           digest.Digest
	UncompressedSize int64
}

type ZstdChunkedFooterData

type ZstdChunkedFooterData struct {
	ManifestType uint64

	Offset             uint64
	LengthCompressed   uint64
	LengthUncompressed uint64

	OffsetTarSplit             uint64
	LengthCompressedTarSplit   uint64
	LengthUncompressedTarSplit uint64
	ChecksumAnnotationTarSplit string // Deprecated: This field is not a part of the footer and not used for any purpose.
}

ZstdChunkedFooterData contains all the data stored in the zstd:chunked footer. This footer exists to make the blobs self-describing, our implementation never reads it: Partial pull security hinges on the TOC digest, and that exists as a layer annotation; so we are relying on the layer annotations anyway, and doing so means we can avoid a round-trip to fetch this binary footer.