dolo

README

dolo

Golang module for file download. Dolo skips downloading if the file exists and has the same content length. Dolo supports an optional callback that notifies on progress.

Exported methods

• NewClient(httpClient *http.Client, notify func(uint64, uint64)) - creates a new dolo client using the provided http.Client (with cookie jar, etc.) and an optional callback to notify on progress. Optional callback takes two values - downloaded bytes and total bytes.

• Download(url *url.URL, dstDir string, overwrite bool) - downloads a resource at the URL to the destination directory (original resource base path would be used for a filename). Overwrite would require download even if the destination file already exists and has the same content length.

Documentation

Index

• Constants
• Variables
• func CopyWithProgress(dst io.Writer, src io.ReadCloser, tpw nod.TotalProgressWriter) error
• func NewResourceContext(u *url.URL, pathParts ...string) *resourceContext
• type Client
  • func NewClient(httpClient *http.Client, opts *ClientOptions) *Client
  • func (dc *Client) Download(u *url.URL, force bool, tpw nod.TotalProgressWriter, pathParts ...string) error
  • func (cl *Client) GetSet(urls []*url.URL, indexSetter IndexSetter, tpw nod.TotalProgressWriter, ...) map[int]error
• type ClientOptions
  • func Defaults() *ClientOptions
• type IndexError
  • func NewIndexError(index int, err error) *IndexError
• type IndexResult
  • func NewIndexResult(index int, result bool) *IndexResult
• type IndexSetter
  • func NewFileIndexSetter(filenames []string) IndexSetter

Constants

const (
	DefaultUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
)

Variables

var DefaultClient = NewClient(http.DefaultClient, Defaults())

Functions

func CopyWithProgress

func CopyWithProgress(dst io.Writer, src io.ReadCloser, tpw nod.TotalProgressWriter) error

func NewResourceContext

func NewResourceContext(u *url.URL, pathParts ...string) *resourceContext

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(httpClient *http.Client, opts *ClientOptions) *Client

func (*Client) Download

func (dc *Client) Download(u *url.URL, force bool, tpw nod.TotalProgressWriter, pathParts ...string) error

Download gets remote resource, attempting to resume existing partial downloads if the previous attempt got interrupted. Download supports nod progress reporting. Download will use provided path parts (see NewResourceContext for file name, directory rules), and if no parts were specified - will download to the nextAvailable working directory and use source URL path base for a filename.

func (*Client) GetSet

func (cl *Client) GetSet(
	urls []*url.URL,
	indexSetter IndexSetter,
	tpw nod.TotalProgressWriter,
	force bool) map[int]error

GetSet downloads URLs and sets them to storage using indexes. E.g. URLs[index] is expected to be received and set by indexSetter(index). GetSet can use provided http.Client for authenticated requests. Finally, it supports reporting progress via provided nod.TotalProgressWriter object (optional).

type ClientOptions

type ClientOptions struct {
	UserAgent          string
	CheckContentLength bool
	ResumeDownloads    bool
}

func Defaults

func Defaults() *ClientOptions

type IndexError

type IndexError struct {
	// contains filtered or unexported fields
}

func NewIndexError

func NewIndexError(index int, err error) *IndexError

type IndexResult

type IndexResult struct {
	// contains filtered or unexported fields
}

func NewIndexResult

func NewIndexResult(index int, result bool) *IndexResult

type IndexSetter

type IndexSetter interface {
	Set(int, io.ReadCloser, chan *IndexResult, chan *IndexError)
	Get(int) (io.ReadCloser, error)
	Exists(int) bool
	Len() int
	IsModifiedAfter(index int, since int64) bool
	CurrentModTime(index int) (int64, error)
}

func NewFileIndexSetter

func NewFileIndexSetter(filenames []string) IndexSetter