paralleldl

README

parallel-dl

parallel-dl is a library for downloading files in parallel.

• Go 1.7 or newer

Features

parallel-dl allows you to control these features:

• HTTP Timeouts
• Maximum number of concurrent requests
• Maximum number of errors before giving up the whole requests
• Maximum number of retries before giving up a request

API

• Download()

That's all !

Examples

opt := &paralleldl.Options{
	Output:           "/path/to/download",
	MaxConcurrents:   2,
	MaxErrorRequests: 1,
	MaxAttempts:      4,
}
client, err := paralleldl.New(opt)
if err != nil {
	log.Fatal(err)
}

lists := []string{
	"http://example.com/file1",
	"http://example.com/file2",
	"http://example.com/file3",
	"http://example.com/file4",
}
errCnt := client.Download(lists)

Minimum Options

opt := &paralleldl.Options{
	Output: "/path/to/download",
}

License


The MIT License

Author

Yoshiki Nakagawa

Documentation

Index

• func SetHTTPClient(client *http.Client)
• type Client
  • func New(opt *Options) (*Client, error)
  • func (c *Client) Download(list []string) int64
• type Options

Functions

func SetHTTPClient

func SetHTTPClient(client *http.Client)

SetHTTPClient overrides the default HTTP client.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client represents http.Client with paralleldl options.

func New

func New(opt *Options) (*Client, error)

New returns a new Client struct.

func (*Client) Download

func (c *Client) Download(list []string) int64

Download downloads the lists resources.

type Options

type Options struct {
	// Output directory
	Output string

	// http.Client's Timeout.
	// A Timeout of zero means no timeout.
	Timeout time.Duration

	// Maximum number of concurrent requests.
	// A MaxConcurrents of zero means no limit of number of requests.
	MaxConcurrents int64
	// Maximum number of errors before giving up the whole requests.
	// A MaxErrorRequests of zero means try until all requests are done.
	MaxErrorRequests int64

	// Maximum number of retries before giving up a request.
	// A MaxAttempts of zero means try until a request is success.
	MaxAttempts int64
}

Options represents Client's parameters.