crawlerdetector

README

About Crawler Detector

Crawler Detector is a Golang package for detecting bots/crawlers/spiders via the user agent.

Installation

Run go get github.com/langaner/crawlerdetector.

Usage

detector := crawlerdetector.New()
isCrawler := detector.IsCrawler(r.Header.Get("User-Agent"))

if isCrawler {
    // Do something
}

Contributing

If you find a bot/spider/crawler user agent that Crawler Detector not detect, please submit a pull request with the regex pattern.

Documentation

Index

• func CrawlersList() []string
• func ExclusionsList() []string
• type CrawlerDetector
  • func New() *CrawlerDetector
  • func (cd *CrawlerDetector) CombineRegexp(patterns []string) string
  • func (cd *CrawlerDetector) GetMatched() []string
  • func (cd *CrawlerDetector) IsCrawler(userAgent string) bool
  • func (cd *CrawlerDetector) IsExclusion(userAgent string) bool
  • func (cd *CrawlerDetector) SetCrawlers(list []string) *CrawlerDetector
  • func (cd *CrawlerDetector) SetExclusions(list []string) *CrawlerDetector

Functions

func CrawlersList

func CrawlersList() []string

CrawlersList is list of crawlers/spiders/bots

func ExclusionsList

func ExclusionsList() []string

ExclusionsList is list of exclusions

Types

type CrawlerDetector

type CrawlerDetector struct {
	Crawlers   []string
	Exclusions []string
	Matched    []string
}

CrawlerDetector is crawler detector structure

func New

func New() *CrawlerDetector

New returns a new initialized CrawlerDetector

func (*CrawlerDetector) CombineRegexp

func (cd *CrawlerDetector) CombineRegexp(patterns []string) string

CombineRegexp is build regex from givement patterns list

func (*CrawlerDetector) GetMatched

func (cd *CrawlerDetector) GetMatched() []string

GetMatched is getter of matched result

func (*CrawlerDetector) IsCrawler

func (cd *CrawlerDetector) IsCrawler(userAgent string) bool

IsCrawler is detect crawlers/spiders/bots by user agent

func (*CrawlerDetector) IsExclusion

func (cd *CrawlerDetector) IsExclusion(userAgent string) bool

IsExclusion is detect exclusion from user agent

func (*CrawlerDetector) SetCrawlers

func (cd *CrawlerDetector) SetCrawlers(list []string) *CrawlerDetector

SetCrawlers is setter for custom crawlers list

func (*CrawlerDetector) SetExclusions

func (cd *CrawlerDetector) SetExclusions(list []string) *CrawlerDetector

SetExclusions is setter for custom exclusions list