crawler

Documentation

Index

• type CrawlerPage
  • func NewCrawlerPage(pager pager.PagerService, database crawler.CrawlerDatabase) CrawlerPage
  • func (p CrawlerPage) Craw(ctx context.Context, uri string, depth uint) ([]string, error)

Types

type CrawlerPage

type CrawlerPage struct {
	// contains filtered or unexported fields
}

CrawlerPage is a implementation to handle with web crawler.

func NewCrawlerPage

func NewCrawlerPage(pager pager.PagerService, database crawler.CrawlerDatabase) CrawlerPage

NewCrawlerPage is a constructor to create a new instance of CrawlerPage.

func (CrawlerPage) Craw

func (p CrawlerPage) Craw(ctx context.Context, uri string, depth uint) ([]string, error)

Craw execute the call to craw pages concurrently and will respect depth param.