downloader

Documentation

Index

• Variables
• func NewLimitedReader(r io.Reader, n int64) io.ReadCloser
• func SHA256(path string) (string, error)
• type BadHTTPCodeError
  • func (e BadHTTPCodeError) Error() string
• type Downloader
  • func New() *Downloader
  • func (d *Downloader) CompareContent() *Downloader
  • func (d *Downloader) Download(ctx context.Context, url string) (bool, error)
  • func (d *Downloader) IfModifiedSince() *Downloader
  • func (d *Downloader) LimitDownloadSize(size int64) *Downloader
  • func (d *Downloader) ToFile(destPath string) *Downloader
  • func (d *Downloader) ValidateOptions() error
  • func (d *Downloader) VerifyHash(hashFunction, hashValue string) *Downloader
  • func (d *Downloader) WithETag(etag string) *Downloader
  • func (d *Downloader) WithETagFile(etagPath string) *Downloader
  • func (d *Downloader) WithETagFn(etagFn func(string) (string, error)) *Downloader
  • func (d *Downloader) WithHTTPClient(client *http.Client) *Downloader
  • func (d *Downloader) WithLastModified() *Downloader
  • func (d *Downloader) WithLogger(logger *logrus.Entry) *Downloader
  • func (d *Downloader) WithMakeDirs(makeDirs bool) *Downloader
  • func (d *Downloader) WithMode(mode os.FileMode) *Downloader
  • func (d *Downloader) WithShelfLife(shelfLife time.Duration) *Downloader
• type NotFoundError
  • func (e NotFoundError) Error() string

Variables

var ErrSizeLimitExceeded = errors.New("size limit exceeded")

Functions

func NewLimitedReader

func NewLimitedReader(r io.Reader, n int64) io.ReadCloser

NewLimitedReader wraps a reader to read up to n bytes before stopping with ErrSizeLimitExceeded. The underlying reader is not closed.

func SHA256

func SHA256(path string) (string, error)

SHA256 returns the hash of the file if possible, empty string otherwise.

Types

type BadHTTPCodeError

type BadHTTPCodeError struct {
	URL  string
	Code int
}

BadHTTPCodeError returned when the server responds with unexpected HTTP code.

func (BadHTTPCodeError) Error

func (e BadHTTPCodeError) Error() string

type Downloader

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader fetches a file from a URL to a destination path, with various options.

func New

func New() *Downloader

New creates a new downloader for the given URL.

func (*Downloader) CompareContent

func (d *Downloader) CompareContent() *Downloader

CompareContent sets the downloader to compare the content after download with the previous file at the destination path. If the content is the same, the old file will stay in place and be reported as up to date. This is useful when there is no other way to check (ETag, Last-Modified...).

func (*Downloader) Download

func (d *Downloader) Download(ctx context.Context, url string) (bool, error)

Download downloads the file from the URL to the destination path. Returns true if the file was downloaded, false if it was already up to date.

func (*Downloader) IfModifiedSince

func (d *Downloader) IfModifiedSince() *Downloader

IfModifiedSince sets the "If-Modified-Since" header to the file's modification time. If the remote resource has not been modified since the given time, the server will respond with a 304.

func (*Downloader) LimitDownloadSize

func (d *Downloader) LimitDownloadSize(size int64) *Downloader

LimitDownloadSize sets the maximum size of the downloaded file, by checking the Content-Length header and monitoring the size again while uncompressing the payload.

func (*Downloader) ToFile

func (d *Downloader) ToFile(destPath string) *Downloader

ToFile sets the destination path for the downloaded file. If a file already exists, its modification time and mode will be used by WithLastModified() and WithMode().

func (*Downloader) ValidateOptions

func (d *Downloader) ValidateOptions() error

ValidateOptions checks that the downloader options are consistent. This is called by Download().

func (*Downloader) VerifyHash

func (d *Downloader) VerifyHash(hashFunction, hashValue string) *Downloader

VerifyHash sets the hash function and value to check the downloaded file against.

func (*Downloader) WithETag

func (d *Downloader) WithETag(etag string) *Downloader

WithETag sets the value to use in the "If-None-Match" header. If an ETag is set, a server will not initiate a download if the remote file matches the ETag.

func (*Downloader) WithETagFile

func (d *Downloader) WithETagFile(etagPath string) *Downloader

WithETagFile sets the path to a file where the ETag will be stored and read.

func (*Downloader) WithETagFn

func (d *Downloader) WithETagFn(etagFn func(string) (string, error)) *Downloader

WithETagFn sets a function whose result will be used in the "If-None-Match" header. The function can compute an ETag from a file's content. The alternative is to store it separately (might be a database identifier or uuid, etc) and use WithETag() to set it directly.

func (*Downloader) WithHTTPClient

func (d *Downloader) WithHTTPClient(client *http.Client) *Downloader

WithHTTPClient sets the http client for the downloader.

func (*Downloader) WithLastModified

func (d *Downloader) WithLastModified() *Downloader

WithLastModified sets the downloader to check the "Last-Modified" header with a HEAD request.

func (*Downloader) WithLogger

func (d *Downloader) WithLogger(logger *logrus.Entry) *Downloader

WithLogger sets the logger for the downloader. If not set, nothing will be logged.

func (*Downloader) WithMakeDirs

func (d *Downloader) WithMakeDirs(makeDirs bool) *Downloader

WithMakeDirs sets whether the downloader should create directories as needed. If not set, the downloader will fail if the destination directory does not exist.

func (*Downloader) WithMode

func (d *Downloader) WithMode(mode os.FileMode) *Downloader

WithMode sets the file mode for the downloaded file. If not set, the file mode will be taken from the destination file, if it exists before the download.

func (*Downloader) WithShelfLife

func (d *Downloader) WithShelfLife(shelfLife time.Duration) *Downloader

WithShelfLife sets the duration after which a file is considered stale, if it has no "Last-Modified" header. If unset, the file will be considered stale by default.

type NotFoundError

type NotFoundError struct {
	URL string
}

NotFoundError returned by Download() when the remote resource is not found.

func (NotFoundError) Error

func (e NotFoundError) Error() string