fulltext

Documentation

Index

• func GetUserAgent() (string, error)
• type Scraper
  • func NewScraper() *Scraper
  • func (s *Scraper) ScrapeUrls(urls ...string) (map[string]WebPage, error)
  • func (s *Scraper) UnredirectUrl(url string) (u string, redirected bool)
• type WebPage

Functions

func GetUserAgent

func GetUserAgent() (string, error)

get user agent returns a valid user agent for use in scraping. in the future the idea is to have it generated at runtime, either by reading from local data or calling a remote api. thus the error return value.

Types

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

func NewScraper

func NewScraper() *Scraper

func (*Scraper) ScrapeUrls

func (s *Scraper) ScrapeUrls(urls ...string) (map[string]WebPage, error)

func (*Scraper) UnredirectUrl

func (s *Scraper) UnredirectUrl(url string) (u string, redirected bool)

type WebPage

type WebPage struct {
	Url        string
	Body       []byte
	Redirected bool
	StatusCode int
}