Documentation
¶
Overview ¶
Package download provides a simple download mechanism that uses the fs.FS container API to implement the actual download. This allows rate control, retries and download management to be separated from the mechanism of the actual download. Downloaders can be provided for http/https, AWS S3 or any other local or cloud storage system for which an fs.FS implementation exists.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Downloaded ¶
Downloaded represents all of the downloads in response to a given request.
type Option ¶
type Option func(*options)
Option is used to configure the behaviour of a newly created Downloader.
func WithNumDownloaders ¶
WithNumDownloaders controls the number of concurrent downloads used. If not specified the default of runtime.GOMAXPROCS(0) is used.
func WithProgress ¶
WithProgress requests that progress messages are sent over the supplid channel. If close is true the progress channel will be closed when the downloader has finished. Close should be set to false if the same channel is shared across multiplied downloader instances.
func WithRateController ¶
func WithRateController(rc *ratecontrol.Controller, retryErr error) Option
WithRateController sets the rate controller to use to enforce rate control. Backoff will be triggered if the supplied error is returned by the container (file.FS) implementation.
type Progress ¶
type Progress struct { // Downloaded is the total number of items downloaded so far. Downloaded int64 // Outstanding is the current size of the input channel for items // yet to be downloaded. Outstanding int64 }
Progress is used to communicate the progress of a download run.
type Request ¶
type Request interface { Requester() string Container() file.FS FileMode() fs.FileMode // FileMode to use for the downloaded contents. Names() []string }
Request represents a request for a list of objects, stored in the same container, to be downloaded.
type Result ¶
type Result struct { // Contents of the download, nil on error. Contents []byte // FileInfo for the downloaded file. FileInfo fs.FileInfo // Name of the downloaded file. Name string // Number of retries that were required to download the file. Retries int // Error encountered during the download. Err error }
Result represents the result of the download for a single object.
type SimpleRequest ¶
SimpleRequest is a simple implementation of Request.
func (SimpleRequest) Container ¶
func (cr SimpleRequest) Container() file.FS
func (SimpleRequest) FileMode ¶
func (cr SimpleRequest) FileMode() fs.FileMode
func (SimpleRequest) Names ¶
func (cr SimpleRequest) Names() []string
func (SimpleRequest) Requester ¶
func (cr SimpleRequest) Requester() string
type T ¶
type T interface { // Run initiates a download run. It reads Requests from the specified // input channel and writes the results of those downloads to the output // channel. Closing the input channel indicates to Run that it should // complete all outstanding download requests. Run will close the output // channel when all requests have been processed. Run(ctx context.Context, input <-chan Request, output chan<- Downloaded) error }
T represents the interface to a downloader that is used to download content.