ztoc

Documentation

Index

• func ExtractFile(r *io.SectionReader, config *FileExtractConfig) ([]byte, error)
• func ExtractFromTarGz(gz string, ztoc *Ztoc, text string) (string, error)
• func GetFileMode(src *FileMetadata) (m os.FileMode)
• func Marshal(ztoc *Ztoc) (io.Reader, ocispec.Descriptor, error)
• func NewGzipZinfo(b []byte)
• func TarProviderGzip(compressedReader *os.File) (io.Reader, error)
• func TarProviderZstd(compressedReader *os.File) (io.Reader, error)
• type BuildOption
  • func WithCompression(algorithm string) BuildOption
• type Builder
  • func NewBuilder(buildToolIdentifier string) *Builder
  • func (b *Builder) BuildZtoc(filename string, span int64, options ...BuildOption) (*Ztoc, error)
  • func (b *Builder) CheckCompressionAlgorithm(algorithm string) bool
  • func (b *Builder) RegisterCompressionAlgorithm(name string, tarProvider TarProvider, zinfoBuilder ZinfoBuilder)
• type CompressionInfo
• type FileExtractConfig
• type FileMetadata
• type MetadataEntry
  • func GetMetadataEntry(ztoc *Ztoc, filename string) (*MetadataEntry, error)
• type TOC
• type TarProvider
• type TocBuilder
  • func NewTocBuilder() TocBuilder
  • func (tb TocBuilder) CheckCompressionAlgorithm(algorithm string) bool
  • func (tb TocBuilder) RegisterTarProvider(algorithm string, provider TarProvider)
  • func (tb TocBuilder) TocFromFile(algorithm, filename string) (TOC, compression.Offset, error)
• type ZinfoBuilder
• type Ztoc
  • func BuildZtocReader(ents []testutil.TarEntry, compressionLevel int, spanSize int64, ...) (*Ztoc, *io.SectionReader, error)
  • func GetZtocFromFile(filename string) (*Ztoc, error)
  • func Unmarshal(serializedZtoc io.Reader) (*Ztoc, error)

Functions

func ExtractFile

func ExtractFile(r *io.SectionReader, config *FileExtractConfig) ([]byte, error)

ExtractFile extracts a file from compressed data (as a reader) and returns the byte data.

func ExtractFromTarGz

func ExtractFromTarGz(gz string, ztoc *Ztoc, text string) (string, error)

ExtractFromTarGz extracts data given a gzip tar file (`gz`) and its `ztoc`.

func GetFileMode

func GetFileMode(src *FileMetadata) (m os.FileMode)

Get file mode for file metadata

func Marshal

func Marshal(ztoc *Ztoc) (io.Reader, ocispec.Descriptor, error)

Marshal serializes Ztoc to its flatbuffers schema and returns a reader along with the descriptor (digest and size only). If not successful, it will return an error.

func NewGzipZinfo

func NewGzipZinfo(b []byte)

NewGzipZinfo is the go implementation of getting "checkpoints" from compressed data.

func TarProviderGzip

func TarProviderGzip(compressedReader *os.File) (io.Reader, error)

TarProviderGzip creates a tar reader from gzip reader.

func TarProviderZstd

func TarProviderZstd(compressedReader *os.File) (io.Reader, error)

TarProviderZstd creates a tar reader from zstd reader.

Types

type BuildOption

type BuildOption func(opt *buildConfig) error

BuildOption specifies a change to `buildConfig` when building a ztoc.

func WithCompression

func WithCompression(algorithm string) BuildOption

WithCompression specifies which compression algorithm is used by the layer.

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

Builder holds a single `TocBuilder` that builds toc, and one `ZinfoBuilder` *per* compression algorithm that builds zinfo. `TocBuilder` is shared by different compression algorithms. Which `ZinfoBuilder` is used depends on the compression algorithm used by the layer.

func NewBuilder

func NewBuilder(buildToolIdentifier string) *Builder

NewBuilder creates a `Builder` used to build ztocs. By default it supports gzip, user can register new compression algorithms by calling `RegisterCompressionAlgorithm`.

func (*Builder) BuildZtoc

func (b *Builder) BuildZtoc(filename string, span int64, options ...BuildOption) (*Ztoc, error)

BuildZtoc builds a `Ztoc` given the filename of a layer blob. By default it assumes the layer is compressed using `gzip`, unless specified via `WithCompression`.

func (*Builder) CheckCompressionAlgorithm

func (b *Builder) CheckCompressionAlgorithm(algorithm string) bool

CheckCompressionAlgorithm checks if a compression algorithm is supported.

The algorithm has to be supported by both (1) `tocBuilder` (straightforward, create a tar reader from the compressed io.reader in compressionFileReader) and (2) `zinfoBuilder` (require zinfo impl, see `compression/gzip_zinfo.go` as an example).

func (*Builder) RegisterCompressionAlgorithm

func (b *Builder) RegisterCompressionAlgorithm(name string, tarProvider TarProvider, zinfoBuilder ZinfoBuilder)

RegisterCompressionAlgorithm supports a new compression algorithm in `ztoc.Builder`.

type CompressionInfo

type CompressionInfo struct {
	MaxSpanID   compression.SpanID //The total number of spans in Ztoc - 1
	SpanDigests []digest.Digest
	Checkpoints []byte
}

CompressionInfo is the "zinfo" part of ztoc including the `Checkpoints` data and other metadata such as all span digests.

type FileExtractConfig

type FileExtractConfig struct {
	UncompressedSize      compression.Offset
	UncompressedOffset    compression.Offset
	Checkpoints           []byte
	CompressedArchiveSize compression.Offset
	MaxSpanID             compression.SpanID
}

FileExtractConfig contains information used to extract a file from compressed data.

type FileMetadata

type FileMetadata struct {
	Name               string
	Type               string
	UncompressedOffset compression.Offset
	UncompressedSize   compression.Offset
	Linkname           string // Target name of link (valid for TypeLink or TypeSymlink)
	Mode               int64  // Permission and mode bits
	UID                int    // User ID of owner
	GID                int    // Group ID of owner
	Uname              string // User name of owner
	Gname              string // Group name of owner

	ModTime  time.Time // Modification time
	Devmajor int64     // Major device number (valid for TypeChar or TypeBlock)
	Devminor int64     // Minor device number (valid for TypeChar or TypeBlock)

	Xattrs map[string]string
}

FileMetadata contains metadata of a file in the compressed data.

type MetadataEntry

type MetadataEntry struct {
	UncompressedSize   compression.Offset
	UncompressedOffset compression.Offset
}

MetadataEntry is used to locate a file based on its metadata.

func GetMetadataEntry

func GetMetadataEntry(ztoc *Ztoc, filename string) (*MetadataEntry, error)

GetMetadataEntry gets MetadataEntry from `ztoc` given a filename.

type TOC

type TOC struct {
	Metadata []FileMetadata
}

TOC is the "ztoc" part of ztoc including metadata of all files in the compressed data (e.g., a gzip tar file).

type TarProvider

type TarProvider func(file *os.File) (io.Reader, error)

TarProvider creates a tar reader from a compressed file reader (e.g., a gzip file reader), which can be used by `TocBuilder` to create `TOC` from it.

type TocBuilder

type TocBuilder struct {
	// contains filtered or unexported fields
}

TocBuilder builds the `TOC` part of a ztoc and works with different compression algorithms (e.g., gzip, zstd) with a registered `TarProvider`.

func NewTocBuilder

func NewTocBuilder() TocBuilder

NewTocBuilder return a `TocBuilder` struct. Users need to call `RegisterTarProvider` to support a specific compression algorithm.

func (TocBuilder) CheckCompressionAlgorithm

func (tb TocBuilder) CheckCompressionAlgorithm(algorithm string) bool

CheckCompressionAlgorithm checks if a compression algorithm is supported.

func (TocBuilder) RegisterTarProvider

func (tb TocBuilder) RegisterTarProvider(algorithm string, provider TarProvider)

RegisterTarProvider adds a TarProvider for a compression algorithm.

func (TocBuilder) TocFromFile

func (tb TocBuilder) TocFromFile(algorithm, filename string) (TOC, compression.Offset, error)

TocFromFile creates a `TOC` given a layer blob filename and the compression algorithm used by the layer.

type ZinfoBuilder

type ZinfoBuilder interface {
	// ZinfoFromFile builds zinfo given a compressed tar filename and span size, and calculate the size of the file.
	ZinfoFromFile(filename string, spanSize int64) (zinfo CompressionInfo, fs compression.Offset, err error)
}

ZinfoBuilder builds the `zinfo` part of a ztoc. This interface should be implemented for each compression algorithm we support.

type Ztoc

type Ztoc struct {
	Version                 string
	BuildToolIdentifier     string
	CompressedArchiveSize   compression.Offset
	UncompressedArchiveSize compression.Offset
	TOC                     TOC
	CompressionInfo         CompressionInfo
}

Ztoc is a table of contents for compressed data which consists 2 parts:

(1). toc (`TOC`): a table of contents containing file metadata and its offset in the decompressed TAR archive. (2). zinfo (`CompressionInfo`): a collection of "checkpoints" of the state of the compression engine at various points in the layer.

func BuildZtocReader

func BuildZtocReader(ents []testutil.TarEntry, compressionLevel int, spanSize int64, opts ...testutil.BuildTarOption) (*Ztoc, *io.SectionReader, error)

BuildZtocReader creates the tar gz file for tar entries. It returns ztoc and io.SectionReader of the file.

func GetZtocFromFile

func GetZtocFromFile(filename string) (*Ztoc, error)

func Unmarshal

func Unmarshal(serializedZtoc io.Reader) (*Ztoc, error)

Unmarshal takes the reader with flatbuffers byte stream and deserializes it ztoc. In case if there's any error situation during deserialization from flatbuffers, there will be an error returned.