scraper

Documentation

Index

• type Scraper
  • func New(cfg config.Config, url *urlpkg.URL, fs afero.Fs) (*Scraper, error)
  • func (sc *Scraper) Cookies() []config.Cookie
  • func (sc *Scraper) Downloader() *download.Download
  • func (sc *Scraper) Start(ctx context.Context) error

Types

type Scraper

type Scraper struct {
	URL *urlpkg.URL // contains the main URL to parse, will be modified in case of a redirect

	Client download.HttpClient
	Fs     afero.Fs // filesystem

	// ETagsDB stores ETags (hashes of file state) for each URL
	ETagsDB *db.DB
	// contains filtered or unexported fields
}

Scraper contains all scraping data, starts the process and handles the concurrency. It includes the logic to decide what URLs to include/exclude and when to stop.

func New

func New(cfg config.Config, url *urlpkg.URL, fs afero.Fs) (*Scraper, error)

New creates a new Scraper instance. nolint: funlen

func (*Scraper) Cookies

func (sc *Scraper) Cookies() []config.Cookie

Cookies returns the current cookies.

func (*Scraper) Downloader

func (sc *Scraper) Downloader() *download.Download

func (*Scraper) Start

func (sc *Scraper) Start(ctx context.Context) error

Start starts the scraping.