Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrBadLength indicates that the server response or an existing file does // not match the expected content length. ErrBadLength = errors.New("bad content length") // ErrBadChecksum indicates that a downloaded file failed to pass checksum // validation. ErrBadChecksum = errors.New("checksum mismatch") // ErrNoFilename indicates that a reasonable filename could not be // automatically determined using the URL or response headers from a server. ErrNoFilename = errors.New("no filename could be determined") // ErrNoTimestamp indicates that a timestamp could not be automatically // determined using the response headers from the remote server. ErrNoTimestamp = errors.New("no timestamp could be determined for the remote file") // ErrFileExists indicates that the destination path already exists. ErrFileExists = errors.New("file exists") )
Functions ¶
func GetBatch ¶
func GetBatch(ctx context.Context, workers int, fileLocs map[string][]string) (<-chan *Response, error)
GetBatch sends multiple HTTP requests and downloads the content of the requested URLs to the given destination directory using the given number of concurrent worker goroutines.
fileLocs is a struct which contains a destination as a directory where multiple urls can write to
The Response for each requested URL is sent through the returned Response channel, as soon as a worker receives a response from the remote server. The Response can then be used to track the progress of the download while it is in progress.
The returned Response channel will be closed, only once all downloads have completed or failed.
If an error occurs during any download, it will be available via call to the associated Response.Err.
For control over HTTP client headers, redirect policy, and other settings, create a Client instead.
func GetTotalURLs ¶
func IsStatusCodeError ¶
IsStatusCodeError returns true if the given error is of type StatusCodeError.
Types ¶
type Client ¶
type Client struct {
// HTTPClient specifies the http.Client which will be used for communicating
// with the remote server during the file transfer.
HTTPClient *http.Client
// UserAgent specifies the User-Agent string which will be set in the
// headers of all requests made by this client.
//
// The user agent string may be overridden in the headers of each request.
UserAgent string
// BufferSize specifies the size in bytes of the buffer that is used for
// transferring all requested files. Larger buffers may result in faster
// throughput but will use more memory and result in less frequent updates
// to the transfer progress statistics. The BufferSize of each request can
// be overridden on each Request object. Default: 32KB.
BufferSize int
}
A Client is a file download client.
Clients are safe for concurrent use by multiple goroutines.
func NewClient ¶
func NewClient() *Client
NewClient returns a new file download Client, using default configuration.
func (*Client) Do ¶
Do sends a file transfer request and returns a file transfer response, following policy (e.g. redirects, cookies, auth) as configured on the client's HTTPClient.
Like http.Get, Do blocks while the transfer is initiated, but returns as soon as the transfer has started transferring in a background goroutine, or if it failed early.
An error is returned via Response.Err if caused by client policy (such as CheckRedirect), or if there was an HTTP protocol or IO error. Response.Err will block the caller until the transfer is completed, successfully or otherwise.
func (*Client) DoBatch ¶
DoBatch executes all the given requests using the given number of concurrent workers. Control is passed back to the caller as soon as the workers are initiated.
If the requested number of workers is less than one, a worker will be created for every request. I.e. all requests will be executed concurrently.
If an error occurs during any of the file transfers it will be accessible via call to the associated Response.Err.
The returned Response channel is closed only after all of the given Requests have completed, successfully or otherwise.
func (*Client) DoChannel ¶
DoChannel executes all requests sent through the given Request channel, one at a time, until it is closed by another goroutine. The caller is blocked until the Request channel is closed and all transfers have completed. All responses are sent through the given Response channel as soon as they are received from the remote servers and can be used to track the progress of each download.
Slow Response receivers will cause a worker to block and therefore delay the start of the transfer for an already initiated connection - potentially causing a server timeout. It is the caller's responsibility to ensure a sufficient buffer size is used for the Response channel to prevent this.
If an error occurs during any of the file transfers it will be accessible via the associated Response.Err function.
type Hook ¶
A Hook is a user provided callback function that can be called by grab at various stages of a requests lifecycle. If a hook returns an error, the associated request is canceled and the same error is returned on the Response object.
Hook functions are called synchronously and should never block unnecessarily. Response methods that block until a download is complete, such as Response.Err, Response.Cancel or Response.Wait will deadlock. To cancel a download from a callback, simply return a non-nil error.
type RateLimiter ¶
RateLimiter is an interface that must be satisfied by any third-party rate limiters that may be used to limit download transfer speeds.
A recommended token bucket implementation can be found at https://godoc.org/golang.org/x/time/rate#Limiter.
type Request ¶
type Request struct {
// HTTPRequest specifies the http.Request to be sent to the remote server to
// initiate a file transfer. It includes request configuration such as URL,
// protocol version, HTTP method, request headers and authentication.
HTTPRequest *http.Request
// NoResume specifies that a partially completed download will be restarted
// without attempting to resume any existing file. If the download is already
// completed in full, it will not be restarted.
NoResume bool
// Path specifies the path where the file transfer will be stored in
// local storage. If Path is empty or a directory, the true Filename will
// be resolved using Content-Disposition headers or the request URL.
//
// An empty string means the transfer will be stored in the current working
// directory.
Filename string
// IgnoreBadStatusCodes specifies that grab should accept any status code in
// the response from the remote server. Otherwise, grab expects the response
// status code to be within the 2XX range (after following redirects).
IgnoreBadStatusCodes bool
// IgnoreRemoteTime specifies that grab should not attempt to set the
// timestamp of the local file to match the remote file.
IgnoreRemoteTime bool
// Size specifies the expected size of the file transfer if known. If the
// server response size does not match, the transfer is cancelled and
// ErrBadLength returned.
Size int64
// BufferSize specifies the size in bytes of the buffer that is used for
// transferring the requested file. Larger buffers may result in faster
// throughput but will use more memory and result in less frequent updates
// to the transfer progress statistics. If a RateLimiter is configured,
// BufferSize should be much lower than the rate limit. Default: 32KB.
BufferSize int
// RateLimiter allows the transfer rate of a download to be limited. The given
// Request.BufferSize determines how frequently the RateLimiter will be
// polled.
RateLimiter RateLimiter
// BeforeCopy is a user provided callback that is called immediately before
// a request starts downloading. If BeforeCopy returns an error, the request
// is cancelled and the same error is returned on the Response object.
BeforeCopy Hook
// AfterCopy is a user provided callback that is called immediately after a
// request has finished downloading, before checksum validation and closure.
// This hook is only called if the transfer was successful. If AfterCopy
// returns an error, the request is canceled and the same error is returned on
// the Response object.
AfterCopy Hook
// contains filtered or unexported fields
}
func NewRequest ¶
NewRequest returns a new file transfer Request suitable for use with Client.Do.
type Response ¶
type Response struct {
// The Request that was submitted to obtain this Response.
Request *Request
// HTTPResponse represents the HTTP response received from an HTTP request.
//
// The response Body should not be used as it will be consumed and closed by
// the grabber.
HTTPResponse *http.Response
// Path specifies the path where the file transfer is stored in local
// storage.
Filename string
// Size specifies the total expected size of the file transfer.
Size int64
// Start specifies the time at which the file transfer started.
Start time.Time
// End specifies the time at which the file transfer completed.
//
// This will return zero until the transfer has completed.
End time.Time
// CanResume specifies that the remote server advertised that it can resume
// previous downloads, as the 'Accept-Ranges: bytes' header is set.
CanResume bool
// DidResume specifies that the file transfer resumed a previously incomplete
// transfer.
DidResume bool
// Done is closed once the transfer is finalized, either successfully or with
// errors. Errors are available via Response.Err
Done chan struct{}
// contains filtered or unexported fields
}
Response represents the response to a download request.
A response may be returned as soon a HTTP response is received from a remote server, but before the body content has started transferring.
All Response method calls are thread-safe
func (*Response) BytesComplete ¶
BytesComplete returns the total number of bytes which have been copied to the destination, including any bytes that were resumed from a previous download.
func (*Response) BytesPerSecond ¶
BytesPerSecond returns the number of bytes transferred in the last second. If the download is already complete, the average bytes/sec for the life of the download is returned.
func (*Response) Duration ¶
Duration returns the duration of a file transfer. If the transfer is in process, the duration will be between now and the start of the transfer. If the transfer is complete, the duration will be between the start and end of the completed transfer process.
func (*Response) ETA ¶
ETA returns the estimated time at which the the download will complete, given the current BytesPerSecond. If the transfer has already completed, the actual end time will be returned.
func (*Response) Err ¶
Err blocks the calling goroutine until the underlying file transfer is completed and returns any error that may have occurred. If the download is already completed, Err returns immediately.
func (*Response) IsComplete ¶
IsComplete returns true if the download has completed. If an error occurred during the download, it can be returned via Err.
type StatusCodeError ¶
type StatusCodeError int
StatusCodeError indicates that the server response had a status code that was not in the 200-299 range (after following any redirects).
func (StatusCodeError) Error ¶
func (err StatusCodeError) Error() string
Source Files