crawler

Documentation

Index

• type Crawler
  • func NewCrawler(instances []*url.URL, numWorkers uint, tokens Tokens, output string) *Crawler
  • func (c *Crawler) Start()
• type SiteData
• type Tokens

Types

type Crawler

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler is used to store crawling configuration

func NewCrawler

func NewCrawler(instances []*url.URL, numWorkers uint, tokens Tokens, output string) *Crawler

NewCrawler creates new `Crawler` instance

func (*Crawler) Start

func (c *Crawler) Start()

Start initializes crawling job

type SiteData

type SiteData struct {
	Link *url.URL
	Body []byte
}

SiteData is used to store web site data

type Tokens

type Tokens map[string]string

Tokens is used to convert JSON tokens into a map type