standard

package
v0.0.0-...-d69ec9c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 14, 2023 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package standard implements the functionality for a non-headless crawler. It uses net/http for making requests and goquery for scraping web page HTML.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Crawler

type Crawler struct {
	*common.Shared
}

Crawler is a standard crawler instance

func New

func New(options *types.CrawlerOptions) (*Crawler, error)

New returns a new standard crawler instance

func (*Crawler) Close

func (c *Crawler) Close() error

Close closes the crawler process

func (*Crawler) Crawl

func (c *Crawler) Crawl(rootURL string) error

Crawl crawls a URL with the specified options

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL