hybrid

Documentation

Overview

Package hybrid implements the functionality for a hybrid-headless crawler. It uses both headless browser and net/http for making requests, and goquery for processing rawand dom-rendered web page HTML.

Index

• func FetchContinueRequest(page *rod.Page, e *proto.FetchRequestPaused) error
• func FetchGetResponseBody(page *rod.Page, e *proto.FetchRequestPaused) ([]byte, error)
• type Crawler
  • func New(options *types.CrawlerOptions) (*Crawler, error)
  • func (c *Crawler) Close() error
  • func (c *Crawler) Crawl(rootURL string) error
• type Hijack
  • func NewHijack(page *rod.Page) *Hijack
  • func (h *Hijack) SetPattern(pattern *proto.FetchRequestPattern)
  • func (h *Hijack) Start(handler HijackHandler) func() error
  • func (h *Hijack) Stop() error
• type HijackHandler

Functions

func FetchContinueRequest

func FetchContinueRequest(page *rod.Page, e *proto.FetchRequestPaused) error

FetchContinueRequest continue request

func FetchGetResponseBody

func FetchGetResponseBody(page *rod.Page, e *proto.FetchRequestPaused) ([]byte, error)

FetchGetResponseBody get request body.

Types

type Crawler

type Crawler struct {
	*common.Shared
	// contains filtered or unexported fields
}

Crawler is a standard crawler instance

func New

func New(options *types.CrawlerOptions) (*Crawler, error)

New returns a new standard crawler instance

func (*Crawler) Close

func (c *Crawler) Close() error

Close closes the crawler process

func (*Crawler) Crawl

func (c *Crawler) Crawl(rootURL string) error

Crawl crawls a URL with the specified options

type Hijack

type Hijack struct {
	// contains filtered or unexported fields
}

Hijack is a hijack handler

func NewHijack

func NewHijack(page *rod.Page) *Hijack

NewHijack create hijack from page.

func (*Hijack) SetPattern

func (h *Hijack) SetPattern(pattern *proto.FetchRequestPattern)

SetPattern set pattern directly

func (*Hijack) Start

func (h *Hijack) Start(handler HijackHandler) func() error

Start hijack.

func (*Hijack) Stop

func (h *Hijack) Stop() error

Stop

type HijackHandler

type HijackHandler = func(e *proto.FetchRequestPaused) error

HijackHandler type