extensions

Documentation

Overview

Package extensions implements various helper addons for qcrawler

Index

• func RandomMobileUserAgent(c *qcrawler.Collector)
• func RandomUserAgent(c *qcrawler.Collector)
• func Referer(c *qcrawler.Collector)
• func URLLengthFilter(c *qcrawler.Collector, URLLengthLimit int)

Functions

func RandomMobileUserAgent

func RandomMobileUserAgent(c *qcrawler.Collector)

RandomMobileUserAgent generates a random MOBILE browser user-agent on every requests

func RandomUserAgent

func RandomUserAgent(c *qcrawler.Collector)

RandomUserAgent generates a random DESKTOP browser user-agent on every requests

func Referer

func Referer(c *qcrawler.Collector)

Referer sets valid Referer HTTP header to requests. Warning: this extension works only if you use Request.Visit from callbacks instead of Collector.Visit.

func URLLengthFilter

func URLLengthFilter(c *qcrawler.Collector, URLLengthLimit int)

URLLengthFilter filters out requests with URLs longer than URLLengthLimit