chonker

README

Chonker

Download large files as parallel chunks using HTTP Range Requests in Go.

Chonker works on Go 1.19, oldstable, and stable releases.

What does Chonker do?

Chonker speeds up downloads from cloud services like Amazon S3 & CloudFront. It does this in two ways.

1. Download small pieces of a file (a.k.a a chunk) using HTTP Range requests.
2. Download chunks in parallel.

Why?

Chonker allows CDN services to cache and serve files even if the entire file is bigger than the individual object cache limit.

It also overcomes the per-connection limit that blob storage services often have by opening connections in parallel.

The Go standard library HTTP Client downloads files as buffered streams of bytes. The Client fetches bytes into a request buffer as fast as it can, and you read bytes from the buffer as fast as you can.

Why is this a problem?

This works great when one beefy connection to an origin server can use the entire available network bandwidth between you and it. Blob file services like Amazon S3 and caching CDNs like Amazon CloudFront impose per-connection limits, but support an almost unlimited number of connections from each client.

If you are downloading a large file from S3, its almost always better to download the file in chunks, using parallel connections.

See the S3 Developer Guide and the CloudFront Developer Guide for more information on cache sizes and parallel GETs.

Use Chonker

Use chonker.Do to fetch a response for a request, or create a "chonky" http.Transport that fetches requests using HTTP Range sub-requests.

Chonker integrates well with Go download libraries. Grab and other download managers can use a http.Client with a "chonky" http.Transport. In turn, Chonker functions accept HTTP clients that could provide automatic retries or detailed logs. See Heimdall or go-retryablehttp for more.

Chonk

Chonk is a Go program that uses the chonker library to download a URL into a local file. Run chonk -h for usage details.

go build -o chonk ./cmd/chonk

./chonk https://example.com

test.sh

test.sh is a BASH shell script that exercises the chonk program with a list of files of varying sizes. Run test.sh -h for usage details.

License

The Chonker cat illustration is from Freepik

Chonker is available under the terms of the MIT license.

See LICENSE for the full license text.

Documentation

Overview

Package chonker implements automatic ranged HTTP requests.

A ranged request is a request that is fetched in chunks using several HTTP requests. Chunks are fetched in separate goroutines by sending HTTP Range requests to the server. Chunks are then concatenated and returned as a single io.Reader. Chunks are chunkSize bytes long. A maximum of workers chunks are fetched concurrently. If the server does not support range requests, the request fails.

Index

• Variables
• func Do(c *http.Client, r *Request) (*http.Response, error)
• func New() *http.Client
• func NewClient(c *http.Client, chunkSize uint64, workers uint) (*http.Client, error)
• func NewRoundTripper(c *http.Client, chunkSize uint64, workers uint) (http.RoundTripper, error)
• type Chunk
  • func Chunks(chunkSize, offset, size uint64) []Chunk
  • func ParseContentRange(s string) (*Chunk, uint64, error)
  • func ParseRange(s string, size uint64) ([]Chunk, error)
  • func (c Chunk) ContentRangeHeader(size uint64) string
  • func (c Chunk) RangeHeader() string
• type Request
  • func NewRequest(method, url string, body io.Reader, chunkSize uint64, workers uint) (*Request, error)
  • func NewRequestWithContext(ctx context.Context, method, url string, body io.Reader, chunkSize uint64, ...) (*Request, error)
  • func (r *Request) WithOpportunisticRange() *Request

Examples

• Do
• NewClient
• NewRequest
• NewRoundTripper

Variables

var (
	ErrInvalidArgument = errors.New(
		"chonker: chunkSize and workers must be greater than zero",
	)
	ErrMultipleRangesUnsupported = errors.New("chonker: multiple ranges not supported")
)

var (
	// ErrRangeNoOverlap is returned by ParseRange if first-byte-pos of
	// all of the byte-range-spec values is greater than the content size.
	ErrRangeNoOverlap = errors.New("chonker: ranges failed to overlap")

	// ErrInvalidRange is returned by ParseRange on invalid input.
	ErrInvalidRange = errors.New("chonker: invalid range")

	// ErrUnsatisfiedRange is returned by ParseContentRange if the range is not satisfied.
	ErrUnsatisfiedRange = errors.New("chonker: unsatisfied range")
)

var ErrRangeUnsupported = errors.New("chonker: server does not support range requests")

var StatsForNerds = metrics.NewSet()

StatsForNerds exposes Prometheus metrics for chonker requests. Metric names are prefixed with "chonker_". Metrics are labeled with request host URLs.

The following metrics are exposed for a request to https://example.com:

chonker_http_requests_fetching{host="example.com"} chonker_http_requests_total{host="example.com"} chonker_http_requests_total{host="example.com",range="false"} chonker_http_request_chunks_fetching{host="example.com",stage="do"} chonker_http_request_chunks_fetching{host="example.com",stage="copy"} chonker_http_request_chunks_total{host="example.com"} chonker_http_request_chunk_duration_seconds{host="example.com"} chonker_http_request_chunk_bytes{host="example.com"}

You can surface these metrics in your application using the metrics.RegisterSet function.

Functions

func Do

func Do(c *http.Client, r *Request) (*http.Response, error)

Do sends an HTTP request and returns an HTTP response, following policy (such as redirects, cookies, auth) as configured on the client. It is a wrapper around http.Client.Do that adds support for ranged requests. A ranged request is a request that is fetched in chunks using several HTTP requests. Chunks are chunkSize bytes long. A maximum of workers chunks are fetched concurrently. HTTP HEAD requests are not fetched in chunks.

Example

req, err := http.NewRequest(http.MethodGet, "http://example.com", nil)
if err != nil {
	panic(err)
}

resp, err := Do(nil, &Request{
	Request:   req,
	chunkSize: 64,
	workers:   8,
})
if err != nil {
	panic(err)
}
defer resp.Body.Close()

func New

func New() *http.Client

New returns a chonker client with default settings. The client fetches four 1MiB chunks concurrently.

func NewClient

func NewClient(c *http.Client, chunkSize uint64, workers uint) (*http.Client, error)

NewClient returns a new http.Client that fetches requests in chunks. The returned client's Transport is a http.RoundTripper from NewRoundTripper.

Example

client, err := NewClient(nil, 64, 8)
if err != nil {
	panic(err)
}

// Use the client.
resp, err := client.Get("http://example.com")
if err != nil {
	panic(err)
}
defer resp.Body.Close()

func NewRoundTripper

func NewRoundTripper(c *http.Client, chunkSize uint64, workers uint) (http.RoundTripper, error)

NewRoundTripper returns a new http.RoundTripper that fetches requests in chunks.

Example

transport, err := NewRoundTripper(nil, 64, 8)
if err != nil {
	panic(err)
}

// Use the transport with a http.Client.
client := &http.Client{Transport: transport}
resp, err := client.Get("http://example.com")
if err != nil {
	panic(err)
}
defer resp.Body.Close()

Types

type Chunk

type Chunk struct {
	Start  uint64
	Length uint64
}

Chunk represents a byte range.

func Chunks

func Chunks(chunkSize, offset, size uint64) []Chunk

Chunks divides the range [offset, size) into chunks of size chunkSize.

func ParseContentRange

func ParseContentRange(s string) (*Chunk, uint64, error)

ParseContentRange parses a Content-Range header string as per RFC 7233. It returns the chunk describing the returned content range, and the size of the content. ErrUnsatisfiedRange is returned if the range is not satisfied.

func ParseRange

func ParseRange(s string, size uint64) ([]Chunk, error)

ParseRange parses a Range header string as per RFC 7233. ErrNoOverlap is returned if none of the ranges fit inside content size. This function is a copy of the parseRange function from the Go standard library net/http/fs.go with minor modifications.

func (Chunk) ContentRangeHeader

func (c Chunk) ContentRangeHeader(size uint64) string

ContentRangeHeader returns a Content-Range header value. Size is the total size of the content. Calling this method on a zero-value Chunk will return an unsatisfied range. For more information on the Content-Range header, see the MDN article on the Content-Range header.

func (Chunk) RangeHeader

func (c Chunk) RangeHeader() string

RangeHeader returns a RangeHeader header value. A zero length is treated as a single byte range. For more information on the Range header, see the MDN article on the Range header.

type Request

type Request struct {
	*http.Request
	// contains filtered or unexported fields
}

Request is a ranged http.Request. It is a wrapper around http.Request that adds support for ranged requests. If the server does not support range requests, the request fails. To succeed even if the server does not support range requests, use WithContinueSansRange.

func NewRequest

func NewRequest(
	method, url string,
	body io.Reader,
	chunkSize uint64,
	workers uint,
) (*Request, error)

NewRequest returns a new Request. See NewRequestWithContext for more information.

Example

req, err := NewRequest(http.MethodGet, "http://example.com", nil, 64, 8)
if err != nil {
	panic(err)
}
fmt.Println(req.URL)

Output:

http://example.com

func NewRequestWithContext

func NewRequestWithContext(
	ctx context.Context,
	method, url string,
	body io.Reader,
	chunkSize uint64, workers uint,
) (*Request, error)

NewRequestWithContext returns a new Request. It is a wrapper around http.NewRequestWithContext that adds support for ranged requests. A ranged request is a request that is fetched in chunks using several HTTP requests. Chunks are chunkSize bytes long. A maximum of workers chunks are fetched concurrently.

func (*Request) WithOpportunisticRange

func (r *Request) WithOpportunisticRange() *Request

WithOpportunisticRange configures r to use ranged sub-requests opportunistically. If the server does not support range requests, the request succeeds anyway.