download

Documentation

Index

• Variables
• type Download
  • func (d *Download) ProcessURL(ctx context.Context, item work.Item) (*url.URL, *work.Result, error)
• type HttpClient
• type SyncCounter
  • func NewHistogram() *SyncCounter
  • func (c *SyncCounter) Increment(code int)
  • func (c *SyncCounter) Map() map[int]int

Variables

var Counters = NewHistogram()

Counters accumulates HTTP response status codes.

Types

type Download

type Download struct {
	Config   config.Config
	ETagsDB  *db.DB
	StartURL *url.URL

	Auth   string
	Client HttpClient
	Fs     afero.Fs // filesystem can be replaced with in-memory filesystem for testing

	Lockdown  *throttle.Throttle // increases sharply when server gives 429 (Too Many Requests) responses, then resets
	LoopDelay *throttle.Throttle // increases only slightly when server gives 429; never decreases
}

Download fetches URLs one by one, sequentially.

func (*Download) ProcessURL

func (d *Download) ProcessURL(ctx context.Context, item work.Item) (*url.URL, *work.Result, error)

type HttpClient

type HttpClient interface {
	Do(req *http.Request) (*http.Response, error)
}

type SyncCounter

type SyncCounter struct {
	// contains filtered or unexported fields
}

func NewHistogram

func NewHistogram() *SyncCounter

func (*SyncCounter) Increment

func (c *SyncCounter) Increment(code int)

Increment adds one to the bucket indicated by code. This is safe for concurrent use.

func (*SyncCounter) Map

func (c *SyncCounter) Map() map[int]int

Map accesses the histogram. This is safe for concurrent use.