fetchers

Documentation

Index

• type Web
  • func NewWeb(keepFragment, keepQuery, stayOnDomain bool) Web
  • func (w Web) Fetch(url string) (content, title string, urls []string, err error)
  • func (w Web) ShouldCrawl(baseURL, href string) bool

Types

type Web

type Web struct {
	// Keep the fragment part of an URL. Example #top"
	KeepFragment bool
	// Keep the query part of an URL. Example ?foo=bar
	KeepQuery bool
	// Is it okay to crawl resources that are stored on another domain?
	StayOnDomain bool
}

A fetcher that fetches resources from the web

func NewWeb

func NewWeb(keepFragment, keepQuery, stayOnDomain bool) Web

Create a new web fetcher.

func (Web) Fetch

func (w Web) Fetch(url string) (content, title string, urls []string, err error)

Returns the title of the crawled URL and a slice of URLs found on that resource

func (Web) ShouldCrawl

func (w Web) ShouldCrawl(baseURL, href string) bool

Determine if we should crawl a link present on the baseURL.