config

Documentation

Index

• func MakeHeaders(headers []string) http.Header
• type Config
  • func (c *Config) GetLaxAge() time.Duration
  • func (c *Config) SensibleDefaults()
• type Cookie

Functions

func MakeHeaders

func MakeHeaders(headers []string) http.Header

Types

type Config

type Config struct {
	Includes []string
	Excludes []string

	Concurrency    int                 // number of concurrent downloads; default 1
	MaxDepth       int                 // download depth, 0 for unlimited
	ImageQuality   images.ImageQuality // image quality from 0 to 100%, 0 to disable reencoding
	RequestTimeout time.Duration       // overall time limit to process each http request
	ConnectTimeout time.Duration       // time limit for connecting to the origin server
	LoopDelay      time.Duration       // fixed value sleep time per request
	LaxAge         time.Duration       // added to origin server's expires timestamp
	Tries          int                 // download attempts, 0 for unlimited

	Directory string
	Username  string
	Password  string

	Cookies   []Cookie
	Header    http.Header
	UserAgent string
}

Config contains the scraper configuration.

func (*Config) GetLaxAge

func (c *Config) GetLaxAge() time.Duration

func (*Config) SensibleDefaults

func (c *Config) SensibleDefaults()

type Cookie

type Cookie struct {
	Name  string `json:"name"`
	Value string `json:"value,omitempty"`

	Expires *time.Time `json:"expires,omitempty"`
}

Cookie represents a cookie, it copies parts of the http.Cookie struct but changes the JSON marshaling to exclude empty fields.