xz

README

Package xz

This Go language package supports the reading and writing of xz compressed streams. It includes also a gxz command for compressing and decompressing data. The package is completely written in Go and doesn't have any dependency on any C code.

The package is currently under development. There might be bugs and APIs are not considered stable. At this time the package cannot compete with the xz tool regarding compression speed and size. The algorithms there have been developed over a long time and are highly optimized. However there are a number of improvements planned and I'm very optimistic about parallel compression and decompression. Stay tuned!

Using the API

The following example program shows how to use the API.

package main

import (
    "bytes"
    "io"
    "log"
    "os"

    "github.com/ulikunitz/xz"
)

func main() {
    const text = "The quick brown fox jumps over the lazy dog.\n"
    var buf bytes.Buffer
    // compress text
    w, err := xz.NewWriter(&buf)
    if err != nil {
        log.Fatalf("xz.NewWriter error %s", err)
    }
    if _, err := io.WriteString(w, text); err != nil {
        log.Fatalf("WriteString error %s", err)
    }
    if err := w.Close(); err != nil {
        log.Fatalf("w.Close error %s", err)
    }
    // decompress buffer and write output to stdout
    r, err := xz.NewReader(&buf)
    if err != nil {
        log.Fatalf("NewReader error %s", err)
    }
    if _, err = io.Copy(os.Stdout, r); err != nil {
        log.Fatalf("io.Copy error %s", err)
    }
}

Documentation

You can find the full documentation at pkg.go.dev.

Using the gxz compression tool

The package includes a gxz command line utility for compression and decompression.

Use following command for installation:

$ go get github.com/ulikunitz/xz/cmd/gxz

To test it call the following command.

$ gxz bigfile

After some time a much smaller file bigfile.xz will replace bigfile. To decompress it use the following command.

$ gxz -d bigfile.xz

Security & Vulnerabilities

The security policy is documented in SECURITY.md.

The software is not affected by the supply chain attack on the original xz implementation, CVE-2024-3094. This implementation doesn't share any files with the original xz implementation and no patches or pull requests are accepted without a review.

All security advisories for this project are published under github.com/ulikunitz/xz/security/advisories.

Documentation

Overview

Package xz supports the compression and decompression of xz files. It supports version 1.0.4 of the specification without the non-LZMA2 filters. See http://tukaani.org/xz/xz-file-format-1.0.4.txt

Example

const text = "The quick brown fox jumps over the lazy dog."
var buf bytes.Buffer

// compress text
w, err := NewWriter(&buf)
if err != nil {
	log.Fatalf("NewWriter error %s", err)
}
if _, err := io.WriteString(w, text); err != nil {
	log.Fatalf("WriteString error %s", err)
}
if err := w.Close(); err != nil {
	log.Fatalf("w.Close error %s", err)
}

// decompress buffer and write result to stdout
r, err := NewReader(&buf)
if err != nil {
	log.Fatalf("NewReader error %s", err)
}
if _, err = io.Copy(os.Stdout, r); err != nil {
	log.Fatalf("io.Copy error %s", err)
}

Output:

The quick brown fox jumps over the lazy dog.

Index

• Constants
• func ValidHeader(data []byte) bool
• type Reader
  • func NewReader(xz io.Reader) (r *Reader, err error)
  • func (r *Reader) Read(p []byte) (n int, err error)
• type ReaderConfig
  • func (c ReaderConfig) NewReader(xz io.Reader) (r *Reader, err error)
  • func (c *ReaderConfig) Verify() error
• type Writer
  • func NewWriter(xz io.Writer) (w *Writer, err error)
  • func (w *Writer) Close() error
  • func (w *Writer) Write(p []byte) (n int, err error)
• type WriterConfig
  • func (c WriterConfig) NewWriter(xz io.Writer) (w *Writer, err error)
  • func (c *WriterConfig) Verify() error

Examples

• Package
• Reader
• Writer

Constants

const (
	None   byte = 0x0
	CRC32  byte = 0x1
	CRC64  byte = 0x4
	SHA256 byte = 0xa
)

Constants for the checksum methods supported by xz.

const HeaderLen = 12

HeaderLen provides the length of the xz file header.

Functions

func ValidHeader

func ValidHeader(data []byte) bool

ValidHeader checks whether data is a correct xz file header. The length of data must be HeaderLen.

Types

type Reader

type Reader struct {
	ReaderConfig
	// contains filtered or unexported fields
}

Reader supports the reading of one or multiple xz streams.

Example

package main

import (
	"bufio"
	"io"
	"log"
	"os"

	"github.com/ulikunitz/xz"
)

func main() {
	f, err := os.Open("fox.xz")
	if err != nil {
		log.Fatalf("os.Open(%q) error %s", "fox.xz", err)
	}
	defer f.Close()
	r, err := xz.NewReader(bufio.NewReader(f))
	if err != nil {
		log.Fatalf("xz.NewReader(f) error %s", err)
	}
	if _, err = io.Copy(os.Stdout, r); err != nil {
		log.Fatalf("io.Copy error %s", err)
	}
}

Output:

The quick brown fox jumps over the lazy dog.

func NewReader

func NewReader(xz io.Reader) (r *Reader, err error)

NewReader creates a new xz reader using the default parameters. The function reads and checks the header of the first XZ stream. The reader will process multiple streams including padding.

func (*Reader) Read

func (r *Reader) Read(p []byte) (n int, err error)

Read reads uncompressed data from the stream.

type ReaderConfig

type ReaderConfig struct {
	DictCap      int
	SingleStream bool
}

ReaderConfig defines the parameters for the xz reader. The SingleStream parameter requests the reader to assume that the underlying stream contains only a single stream.

func (ReaderConfig) NewReader

func (c ReaderConfig) NewReader(xz io.Reader) (r *Reader, err error)

NewReader creates an xz stream reader. The created reader will be able to process multiple streams and padding unless a SingleStream has been set in the reader configuration c.

func (*ReaderConfig) Verify

func (c *ReaderConfig) Verify() error

Verify checks the reader parameters for Validity. Zero values will be replaced by default values.

type Writer

type Writer struct {
	WriterConfig
	// contains filtered or unexported fields
}

Writer compresses data written to it. It is an io.WriteCloser.

Example

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/ulikunitz/xz"
)

func main() {
	f, err := os.Create("example.xz")
	if err != nil {
		log.Fatalf("os.Open(%q) error %s", "example.xz", err)
	}
	defer f.Close()
	w, err := xz.NewWriter(f)
	if err != nil {
		log.Fatalf("xz.NewWriter(f) error %s", err)
	}
	defer w.Close()
	_, err = fmt.Fprintln(w, "The brown fox jumps over the lazy dog.")
	if err != nil {
		log.Fatalf("fmt.Fprintln error %s", err)
	}
	if err = w.Close(); err != nil {
		log.Fatalf("w.Close() error %s", err)
	}
}

func NewWriter

func NewWriter(xz io.Writer) (w *Writer, err error)

NewWriter creates a new xz writer using default parameters.

func (*Writer) Close

func (w *Writer) Close() error

Close closes the writer and adds the footer to the Writer. Close doesn't close the underlying writer.

func (*Writer) Write

func (w *Writer) Write(p []byte) (n int, err error)

Write compresses the uncompressed data provided.

type WriterConfig

type WriterConfig struct {
	Properties *lzma.Properties
	DictCap    int
	BufSize    int
	BlockSize  int64
	// checksum method: CRC32, CRC64 or SHA256 (default: CRC64)
	CheckSum byte
	// Forces NoChecksum (default: false)
	NoCheckSum bool
	// match algorithm
	Matcher lzma.MatchAlgorithm
}

WriterConfig describe the parameters for an xz writer.

func (WriterConfig) NewWriter

func (c WriterConfig) NewWriter(xz io.Writer) (w *Writer, err error)

NewWriter creates a new Writer using the given configuration parameters.

func (*WriterConfig) Verify

func (c *WriterConfig) Verify() error

Verify checks the configuration for errors. Zero values will be replaced by default values.