services

Documentation

Index

• type Crawler
  • func NewCrawler(urlFetcher UrlFetcher, depth int, logger *log.Logger) *Crawler
  • func (c *Crawler) GenerateSitemap(urlToFetch *url.URL, timeout time.Duration) datatypes.Sitemap
• type UrlFetcher

Types

type Crawler

type Crawler struct {
	LinkList *datatypes.SitemapList
	// contains filtered or unexported fields
}

Crawler represents a recursive url fetcher

func NewCrawler

func NewCrawler(urlFetcher UrlFetcher, depth int, logger *log.Logger) *Crawler

NewCrawler creates a new Crawler instance

func (*Crawler) GenerateSitemap

func (c *Crawler) GenerateSitemap(urlToFetch *url.URL, timeout time.Duration) datatypes.Sitemap

GenerateSitemap fetch a url and generate a sitemap datastructure

type UrlFetcher

type UrlFetcher interface {
	ParseUrl(newUrl string) ([]string, []string, error)
}

UrlFetcher is an interface for URL Fetching