asyncdl

README

asyncdl - the asynchronous donwloader

Documentation

Overview

Package adownload provides a simple download Manager for asynchronous download of a list of URLs.

Index

• func Download(ctx context.Context, fsa fsadapter.FS, subdir string, urls []string, ...) error
• func DownloadToPath(ctx context.Context, zipOrDir string, subdir string, urls []string, ...) error
• type Manager
  • func New(fsa fsadapter.FS, opts ...Option) *Manager
  • func NewWithPath(zipOrDir string, opts ...Option) (*Manager, error)
  • func (m *Manager) Close() error
  • func (m *Manager) Download(ctx context.Context, dir string, urls []string) error
• type Option
  • func IgnoreHTTPErrors(isEnabled bool) Option
  • func NumWorkers(n int) Option
  • func WithClient(cl *http.Client) Option

Functions

func Download

func Download(ctx context.Context, fsa fsadapter.FS, subdir string, urls []string, opts ...Option) error

Download downloads files given the list of urls to the subdirectory within the filesystem wrapped by fs adapter fsa. URLs must point to files, otherwise the behaviour is undefined. It does not support the recursive download.

func DownloadToPath

func DownloadToPath(ctx context.Context, zipOrDir string, subdir string, urls []string, opts ...Option) error

DownloadToPath downloads urls into a subdirectory subdir of a ZIP file or directory specified by zipOrDir. See Download for more details.

Types

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager is the download manager.

func New

func New(fsa fsadapter.FS, opts ...Option) *Manager

New creates a new download Manager.

func NewWithPath

func NewWithPath(zipOrDir string, opts ...Option) (*Manager, error)

NewWithPath creates a download Manager which initialises a file system adapter over the ZIP file or Directory specified in zipOrDir. It must be closed with [Close] to ensure that all buffers are flushed to disk.

func (*Manager) Close

func (m *Manager) Close() error

Close closes the underlying filesystem adapter, if necessary. If the Manager wasn't initialised with NewWithPath then calling Close is noop.

func (*Manager) Download

func (m *Manager) Download(ctx context.Context, dir string, urls []string) error

Download downloads the files and saves them to the dir directory within the file system adapter fsa. It spawns numWorker goroutines for getting the files. It will call fetchFn for each url.

type Option

type Option func(m *Manager)

Option is the download manager option.

func IgnoreHTTPErrors

func IgnoreHTTPErrors(isEnabled bool) Option

IgnoreHTTPErrors allows to ignore HTTP errors (enabled by default)

func NumWorkers

func NumWorkers(n int) Option

NumWorkers allows to set the number of goroutines that will download the files.

func WithClient

func WithClient(cl *http.Client) Option

WithClient allows to specify a custom HTTP client to use.