sprinter

Documentation

Overview

Package sprinter implements our fast web crawler.

The sprinter package is a concurrent webquery engine.

Index

• Constants
• Variables
• type Crawler
  • func NewCrawler(storage storage.Storage, buffer containers.Container) (c *Crawler, err error)
  • func (c *Crawler) Crawl(uri string) (err error)
  • func (c *Crawler) CrawlSequential(uri string) (err error)

Constants

const (
	RobotsSize = 2000
)

Variables

var (
	ErrInvalidParameters = errors.New("invalid parameters for crawler")
	IgnoredWords         = []string{"the"}
)

var (
	Info  *log.Logger
	Error *log.Logger
)

Types

type Crawler

type Crawler struct {
	MaxRequests           int // The max number of requests that can be handled in total.
	MaxConcurrentRequests int // The max number of requests that can be handled concurrently.

	Verbose bool
	// contains filtered or unexported fields
}

func NewCrawler

func NewCrawler(storage storage.Storage, buffer containers.Container) (c *Crawler, err error)

Create a new Crawler object with the specified storage.Storage and link buffer.

func (*Crawler) Crawl

func (c *Crawler) Crawl(uri string) (err error)

Start at the URI and crawl from there.

func (*Crawler) CrawlSequential

func (c *Crawler) CrawlSequential(uri string) (err error)

A somewhat easier function to crawl sequentially. See also the Crawl function.