crawler

Documentation

Index

• type Config
• type Crawler
  • func NewCrawler(f fetcher.Fetcher, l *logr.Logger, cfg Config) *Crawler
  • func (c *Crawler) Crawl(url string) ([]string, error)

Types

type Config

type Config struct {
	// Whether or not to follow external links on a scraped page
	FollowExternalLinks bool
}

Config represents crawler configuration

type Crawler

type Crawler struct {
	fetcher.Fetcher
	*logr.Logger
	Config
}

Crawler holds the crawler data structure

func NewCrawler

func NewCrawler(f fetcher.Fetcher, l *logr.Logger, cfg Config) *Crawler

NewCrawler builds a `Crawler` object

func (*Crawler) Crawl

func (c *Crawler) Crawl(url string) ([]string, error)

Crawl receivers a string `url` and it will return the links ([]string) found