crawl

Documentation

Index

• type Scraper
  • func New() Scraper

Types

type Scraper

type Scraper interface {
	// Scrape crawls a webpage obtaining content from the
	// website by the given URL.
	// A http.MethodGet request is made to the URL and the page is
	// analysed for the element by the given selector providing
	// a 200 response is returned.
	//
	// Returns errors.INVALID if the URL failed to parse, the client
	// could not make request or if the status code is not 200.
	// Returns errors.INTERNAL if the document could not be parsed
	// body could not be read.
	Scrape(url, selector string) (string, error)
}

Scraper defines the method used for crawling a singular URL.

func New

func New() Scraper

New creates a new scraper with a custom http.Client and cookie Jar. Returns errors.INTERNAL if the jar could not be created.