passive

Documentation

Overview

Package passive implements the functionality for a non-headless crawler. It uses net/http for making requests and goquery for scraping web page HTML.

Index

• Variables
• type Crawler
  • func New(options *types.CrawlerOptions) (*Crawler, error)
  • func (c *Crawler) Close() error
  • func (c *Crawler) Crawl(rootURL string) error

Variables

var Sources = map[string]source.Source{
	"waybackarchive": &waybackarchive.Source{},
	"commoncrawl":    &commoncrawl.Source{},
	"alienvault":     &alienvault.Source{},
}

Types

type Crawler

type Crawler struct {
	*common.Shared
	// contains filtered or unexported fields
}

Crawler is a passive crawler instance

func New

func New(options *types.CrawlerOptions) (*Crawler, error)

New returns a new passive crawler instance

func (*Crawler) Close

func (c *Crawler) Close() error

Close closes the crawler process

func (*Crawler) Crawl

func (c *Crawler) Crawl(rootURL string) error

Crawl crawls a URL with the specified options