extensions

Documentation

Overview

Package extensions implements various helper addons for Krkn

Index

• func RandomUserAgent(c *krkn.Collector)
• func Referrer(c *krkn.Collector)
• func URLLengthFilter(c *krkn.Collector, URLLengthLimit int)

Functions

func RandomUserAgent

func RandomUserAgent(c *krkn.Collector)

RandomUserAgent generates a random browser user agent on every request

func Referrer

func Referrer(c *krkn.Collector)

Referrer sets valid Referrer HTTP header to requests. Warning: this extension works only if you use Request.Visit from callbacks instead of Collector.Visit.

func URLLengthFilter

func URLLengthFilter(c *krkn.Collector, URLLengthLimit int)

URLLengthFilter filters out requests with URLs longer than URLLengthLimit