gofetch

README

Gofetch

Go library to download files from the internerds using Go 1.7 or greater.

Features

• Resumes downloads if interrupted.
• Allows parallel downloading of a single file by requesting multiple data chunks at once over HTTP.
• Reports download progress through a Go channel if indicated to do so.
• Supports file integrity verification if a checksum is provided.
• Supports ETags, skipping downloading a file if it hasn't changed on the server.
• Can be combined with https://github.com/cenkalti/backoff to support retrying with exponential back-off

Gotchas

When downloading file chunks concurrently, you may encounter some issues:

• Servers may limit the number of concurrent connections you have open against them
• Servers may accept the connections but will not send anything, in this case the default HTTP client will timeout for you. If you provide your own client, make sure it has proper timeouts or your connection will block for several seconds, depending on your operating system network timeouts.

Example

package main

import (
	"fmt"
	"io"
	"os"

	"github.com/c4milo/gofetch"
)

func main() {
	gf := gofetch.New(
		gofetch.WithDestDir(os.TempDir()),
		gofetch.WithConcurrency(10),
		gofetch.WithETag(),
	)

	progressCh := make(chan gofetch.ProgressReport)

	var myFile *os.File
	go func() {
		var err error
		myFile, err = gf.Fetch(
			"http://releases.ubuntu.com/15.10/ubuntu-15.10-server-amd64.iso",
			progressCh)
		if err != nil {
			panic(err)
		}
	}()

	// pogressCh is close by gofetch once a download finishes
	var totalWritten int64
	for p := range progressCh {
		// p.WrittenBytes does not accumulate, it represents the chunk size written
		// in the current operation.
		totalWritten += p.WrittenBytes
		fmt.Printf("%d of %d\n", totalWritten, p.Total)
	}

	destFile, err := os.Create("/tmp/ubuntu-15.10-server-amd64.iso")
	if err != nil {
		panic(err)
	}

	defer func() {
		if err := destFile.Close(); err != nil {
			panic(err)
		}
	}()

	if _, err := io.Copy(destFile, myFile); err != nil {
		panic(err)
	}
}

Documentation

Overview

Example

package main

import (
	"fmt"
	"os"

	"github.com/c4milo/gofetch"
)

func main() {
	gf := gofetch.New(
		gofetch.WithDestDir("/tmp"),
		gofetch.WithConcurrency(100),
		gofetch.WithETag(),
		gofetch.WithChecksum("sha256", "29a8b9009509b39d542ecb229787cdf48f05e739a932289de9e9858d7c487c80"),
	)

	progressCh := make(chan gofetch.ProgressReport)
	doneCh := make(chan struct{})

	var myFile *os.File
	go func() {
		defer close(doneCh)

		var err error
		myFile, err = gf.Fetch(
			"http://releases.ubuntu.com/16.04.1/ubuntu-16.04.1-server-amd64.iso",
			progressCh)
		if err != nil {
			panic(err)
		}
	}()

	// pogressCh is close by gofetch once a download finishes
	var totalWritten int64
	for p := range progressCh {
		// p.WrittenBytes does not accumulate, it represents the chunk size written
		// in the current operation.
		totalWritten += p.WrittenBytes
		fmt.Printf("\r%d of %d bytes", totalWritten, p.Total)
	}

	<-doneCh
	fmt.Printf("\nFile saved at %q\n", myFile.Name())
}

Index

• type Fetcher
  • func New(opts ...Option) *Fetcher
  • func (gf *Fetcher) Fetch(url string, progressCh chan<- ProgressReport) (*os.File, error)
• type Option
  • func WithChecksum(alg, value string) Option
  • func WithConcurrency(c int) Option
  • func WithDestDir(dir string) Option
  • func WithETag() Option
  • func WithHTTPClient(c *http.Client) Option
• type ProgressReport

Examples

• Package

Types

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher represents an instance of gofetch, holding global configuration options.

func New

func New(opts ...Option) *Fetcher

New creates a new instance of goFetch with the given options.

func (*Fetcher) Fetch

func (gf *Fetcher) Fetch(url string, progressCh chan<- ProgressReport) (*os.File, error)

Fetch downloads content from the provided URL. It supports resuming and parallelizing downloads while being very memory efficient.

type Option

type Option func(*Fetcher)

Option as explained in http://commandcenter.blogspot.com/2014/01/self-referential-functions-and-design.html

func WithChecksum

func WithChecksum(alg, value string) Option

WithChecksum verifies the file once it is fully downloaded using the provided hash and expected value.

func WithConcurrency

func WithConcurrency(c int) Option

WithConcurrency allows you to set the number of goroutines used to download a specific file. By default it is set to 1.

func WithDestDir

func WithDestDir(dir string) Option

WithDestDir allows you to set the destination directory for the downloaded files. By default it is set to: ./

func WithETag

func WithETag() Option

WithETag enables ETag support, meaning that if an already downloaded file is currently on disk and matches the ETag value returned by the server, it will not be downloaded again. By default it is set to false. Be aware that different servers, serving the same file, are likely to return different ETag values, causing the file to be re-downloaded, even though it might already exist on disk.

func WithHTTPClient

func WithHTTPClient(c *http.Client) Option

WithHTTPClient allows to provide a custom HTTP client. By default a HTTP client with support for read/write timeouts is used.

type ProgressReport

type ProgressReport struct {
	// Total length in bytes of the file being downloaded
	Total int64
	// Written bytes to disk on a write by write basis. It does not accumulate.
	WrittenBytes int64
}

ProgressReport represents the current download progress of a given file