httpcompression

README

Golang HTTP server middleware for gzip/brotli/zstandard compression

This is a small Go package which wraps HTTP handlers to transparently compress response bodies using zstd, brotli, gzip or deflate - for clients which support them. Although it's usually simpler to leave that to a reverse proxy (like nginx or Varnish), this package is useful when that is undesirable. In addition, this package allows users to extend it by plugging in third-party or custom compression encoders.

Note: This package was recently forked from the dead NYTimes/gziphandler. Maintaining drop-in compatibility is not a goal of this fork, as the scope of this fork is significantly wider than the original package.

⚠ As we have not reached 1.0 yet, API is still subject to changes.

Features

• gzip, deflate, brotli, and zstd compression by default, alternate (faster) gzip, zstd implementations are optional
• Apply compression only if response body size is greater than a threshold
• Apply compression only to a allowlist/denylist of MIME content types
• Define encoding priority (e.g. give brotli a higher priority than gzip)
• Control whether the client or the server defines the encoder priority
• Plug in third-party/custom compression schemes or implementations
• Custom dictionary compression for zstd and deflate
• Low memory alliocations via transparent encoder reuse

Demo

While no dedicated demo exists, the demo website for regexp2go internally uses httpcompression to transparently compress responses.

Install

go get github.com/CAFxX/httpcompression

Usage

Call httpcompression.DefaultAdapter to get an adapter that can be used to wrap any handler (an object which implements the http.Handler interface), to transparently provide response body compression. Note that httpcompression automatically compresses using Zstandard, Brotli, Deflate, and Gzip depending on the capabilities of the client (Accept-Encoding) and the configuration of this handler (by default, Zstandard, Brotli and gzip are all enabled and, conditional on client support, used in that order of preference).

As a simple example:

package main

import (
    "io"
    "net/http"
    "github.com/CAFxX/httpcompression"
)

func main() {
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain")
        io.WriteString(w, "Hello, World")
    })
    compress, _ := httpcompression.DefaultAdapter() // Use the default configuration
    http.Handle("/", compress(handler))
    http.ListenAndServe("0.0.0.0:8080", nil)
}

Pluggable compressors

It is possible to use custom compressor implementations by specifying a CompressorProvider for each of the encodings the adapter should support. This also allows to support arbitrary Content-Encoding schemes (e.g. lzma, or zstd with a static dictionary - see the examples).

pgz, err := httpcompression.Compressor("gzip", 0, pgzip.New(pgzip.Options{Level: 6}))
if err != nil {
    log.Fatal(err)
}
compress, err := httpcompression.Adapter(
    // use klauspost/pgzip as compressor for the "gzip" content-encoding, with priority 0
    pgz,
)
if err != nil {
    log.Fatal(err)
}
http.Handle("/", compress(handler))

The contrib/ directory contains a number of bundled implementations that are ready for use:

Content-Encoding	Provider package	Implementation package	Notes	Dictionary	Go/cgo	Default	IANA registry
deflate	contrib/compress/zlib	compress/zlib	Slower than klauspost/zlib	Yes	Go	Yes	Yes
deflate	contrib/klauspost/zlib	github.com/klauspost/compress/zlib		Yes	Go	No	Yes
gzip	contrib/compress/gzip	compress/gzip	Slower than klauspost/gzip	No	Go	Yes	Yes
gzip	contrib/klauspost/gzip	github.com/klauspost/compress/gzip		No	Go	No	Yes
gzip	contrib/klauspost/pgzip	github.com/klauspost/pgzip	Parallel compression	No	Go	No	Yes
zstd	contrib/klauspost/zstd	github.com/klauspost/compress/zstd		Yes	Go	Yes	Yes
zstd	contrib/valyala/gozstd	github.com/valyala/gozstd	Slower than klauspost/zstd	Yes	cgo	No	Yes
brotli	contrib/andybalholm/brotli	github.com/andybalholm/brotli	Slower than google/brotli	No	Go	Yes	Yes
brotli	contrib/google/cbrotli	github.com/google/brotli	Requires brotli libraries to be installed	No	cgo	No	Yes
lz4	contrib/pierrec/lz4	github.com/pierrec/lz4/v4		No	Go	No	No
xz	contrib/ulikunitz/xz	github.com/ulikunitz/xz		No	Go	No	No

Framework integration

In addition to the default support for net/http, httpcompression provides adapters for the following web frameworks:

Framework	Adapter
github.com/gofiber/fiber/v2	contrib/gofiber/fiber/v2
github.com/labstack/echo	contrib/labstack/echo
github.com/gin-gonic/gin	contrib/gin-gonic/gin

Benchmark

See the benchmark results to get an idea of the relative performance and compression efficiency of gzip, brotli and zstd in the current implementation.

TODO

• Add dictionary support to brotli (zstd and deflate already support it, gzip does not allow dictionaries)
• Allow to choose dictionary based on content-type
• Provide additional implementations based on the bindings to the original native implementations
• Add compressed payload caching (if the same payload has already been compressed and is present in the cache, skip compression)
• Add write buffering (compress larger chunks at once)
• Add decompression (if the payload is already compressed but the client supports better algorithms, or does not support a certain algorithm)
• Add other, non-standardized content encodings (lzma/lzma2/xz, snappy, bzip2, etc.)
• Dynamically tune MinSize (and possibly also ContentTypes, level/quality, ...)
• Automatically generate and serve dictionaries

License

Apache 2.0.

Documentation

Overview

Example

package main

import (
	"log"
	"net/http"

	"github.com/CAFxX/httpcompression"
)

func main() {
	// Create a compression adapter with default configuration
	compress, err := httpcompression.DefaultAdapter()
	if err != nil {
		log.Fatal(err)
	}
	// Define your handler, and apply the compression adapter.
	http.Handle("/", compress(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("Hello world!"))
	})))
	// ...
}

Index

• Constants
• func Adapter(opts ...Option) (func(http.Handler) http.Handler, error)
• func DefaultAdapter(opts ...Option) (func(http.Handler) http.Handler, error)
• type CompressorProvider
  • func NewDefaultGzipCompressor(level int) (CompressorProvider, error)
• type Flusher
• type Option
  • func BrotliCompressionLevel(level int) Option
  • func BrotliCompressor(b CompressorProvider) Option
  • func Compressor(contentEncoding string, priority int, compressor CompressorProvider) Option
  • func ContentTypes(types []string, blacklist bool) Option
  • func DeflateCompressionLevel(level int) Option
  • func DeflateCompressor(g CompressorProvider) Option
  • func GzipCompressionLevel(level int) Option
  • func GzipCompressor(g CompressorProvider) Option
  • func MinSize(size int) Option
  • func Prefer(prefer PreferType) Option
  • func ZstandardCompressor(b CompressorProvider) Option
• type PreferType

Examples

• Package

Constants

const (
	// DefaultMinSize is the default minimum response body size for which we enable compression.
	//
	// 200 is a somewhat arbitrary number; in experiments compressing short text/markup-like sequences
	// with different compressors we saw that sequences shorter that ~180 the output generated by the
	// compressor would sometime be larger than the input.
	// This default may change between versions.
	// In general there can be no one-size-fits-all value: you will want to measure if a different
	// minimum size improves end-to-end performance for your workloads.
	DefaultMinSize = 200
)

Functions

func Adapter

func Adapter(opts ...Option) (func(http.Handler) http.Handler, error)

Adapter returns a HTTP handler wrapping function (a.k.a. middleware) which can be used to wrap an HTTP handler to transparently compress the response body if the client supports it (via the Accept-Encoding header). It is possible to pass one or more options to modify the middleware configuration. If no options are provided, no compressors are enabled and therefore the adapter is a no-op. An error will be returned if invalid options are given.

func DefaultAdapter

func DefaultAdapter(opts ...Option) (func(http.Handler) http.Handler, error)

DefaultAdapter is like Adapter, but it includes sane defaults for general usage. Currently the defaults enable gzip and brotli compression, and set a minimum body size of 200 bytes. The provided opts override the defaults. The defaults are not guaranteed to remain constant over time: if you want to avoid this use Adapter directly.

Types

type CompressorProvider

type CompressorProvider interface {
	// Get returns a writer that writes compressed output to the supplied parent io.Writer.
	// Callers of Get() must ensure to always call Close() when the compressor is not needed
	// anymore. Callers of Close() must also ensure to not use the io.WriteCloser once Close()
	// is called.
	// Implementations of CompressorProvider are allowed to recycle the compressor (e.g. put the
	// WriteCloser in a pool to be reused by a later call to Get) when Close() is called.
	// The returned io.WriteCloser can optionally implement the Flusher interface if it is
	// able to flush data buffered internally.
	Get(parent io.Writer) (compressor io.WriteCloser)
}

CompressorProvider is the interface for compression implementations.

func NewDefaultGzipCompressor

func NewDefaultGzipCompressor(level int) (CompressorProvider, error)

type Flusher

type Flusher interface {
	// Flush flushes the data buffered internally by the Writer.
	// Implementations of Flush do not need to internally flush the parent Writer.
	Flush() error
}

Flusher is an optional interface that can be implemented by the compressors returned by CompressorProvider.Get().

type Option

type Option func(c *config) error

Option can be passed to Handler to control its configuration.

func BrotliCompressionLevel

func BrotliCompressionLevel(level int) Option

BrotliCompressionLevel is an option that controls the Brotli compression level to be used when compressing payloads. The default is 3 (the same default used in the reference brotli C implementation).

func BrotliCompressor

func BrotliCompressor(b CompressorProvider) Option

BrotliCompressor is an option to specify a custom compressor factory for Brotli.

func Compressor

func Compressor(contentEncoding string, priority int, compressor CompressorProvider) Option

Compressor returns an Option that sets the CompressorProvider for a specific Content-Encoding. If multiple CompressorProviders are set for the same Content-Encoding, the last one is used. If compressor is nil, it disables the specified Content-Encoding. Priority is used to specify the priority of the Content-Encoding. A higher number means higher priority. See PreferType to understand how priority is used to select the Content-Encoding for a specific request.

func ContentTypes

func ContentTypes(types []string, blacklist bool) Option

ContentTypes specifies a list of content types to compare the Content-Type header to before compressing. If none match, and blacklist is false, the response will be returned as-is.

Content types are compared in a case-insensitive, whitespace-ignored manner.

A MIME type without any other directive will match a content type that has the same MIME type, regardless of that content type's other directives. I.e., "text/html" will match both "text/html" and "text/html; charset=utf-8".

A MIME type with any other directive will only match a content type that has the same MIME type and other directives. I.e., "text/html; charset=utf-8" will only match "text/html; charset=utf-8".

If blacklist is true then only content types that do not match the provided list of types are compressed. If blacklist is false, only content types that match the provided list are compressed.

By default, responses are compressed regardless of Content-Type.

func DeflateCompressionLevel

func DeflateCompressionLevel(level int) Option

DeflateCompressionLevel is an option that controls the Deflate compression level to be used when compressing payloads. The default is flate.DefaultCompression.

func DeflateCompressor

func DeflateCompressor(g CompressorProvider) Option

DeflateCompressor is an option to specify a custom compressor factory for Deflate.

func GzipCompressionLevel

func GzipCompressionLevel(level int) Option

GzipCompressionLevel is an option that controls the Gzip compression level to be used when compressing payloads. The default is gzip.DefaultCompression.

func GzipCompressor

func GzipCompressor(g CompressorProvider) Option

GzipCompressor is an option to specify a custom compressor factory for Gzip.

func MinSize

func MinSize(size int) Option

MinSize is an option that controls the minimum size of payloads that should be compressed. The default is DefaultMinSize.

func Prefer

func Prefer(prefer PreferType) Option

Prefer controls the behavior of the middleware in case both Gzip and Brotli can be used to compress a response (i.e. in case the client supports both encodings, and the MIME type of the response is allowed for both encodings). See the comments on the PreferType constants for the supported values.

func ZstandardCompressor

func ZstandardCompressor(b CompressorProvider) Option

ZstandardCompressor is an option to specify a custom compressor factory for Zstandard.

type PreferType

type PreferType byte

PreferType allows to control the choice of compression algorithm when multiple algorithms are allowed by both client and server.

const (
	// PreferServer prefers compressors in the order specified on the server.
	// If two or more compressors have the same priority on the server, the client preference is taken into consideration.
	// If both server and client do no specify a preference between two or more compressors, the order is determined by the name of the encoding.
	// PreferServer is the default.
	PreferServer PreferType = iota

	// PreferClient prefers compressors in the order specified by the client.
	// If two or more compressors have the same priority according to the client, the server priority is taken into consideration.
	// If both server and client do no specify a preference between two or more compressors, the order is determined by the name of the encoding.
	PreferClient
)