botdetect

README

Botdetect

botdetect is a small program that reads IPs and an URI Path on STDIN in the form IP1|IP2|PATH, e.g. "192.168.12.3|34.88.27.218|/index.html" and calculates which IP produces too many requests where the ratio between HTML requests an asset requests (images, CSS, Javascript, fonts) leans too far in favour of HTML.

Assuming that bad bots usually don't load assets, this is a relatively straight-forward way of detecting bad bots.

Please note that this is only a starting point for a bot blocker that you can use in production. You will at least need to add a way to whitelist IPs, hostnames or entire networks. Otherwise the program will block everything that exceeds the limits, including good bots like Google or Bing or any IPs that you need to be able to access your website.

Integration into Apache

botdetect can be easily integrated into the apache webserver through mod_rewrite's "prg" interface that lets an external program decide the outcome of a RewriteCond operation:

LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so

RewriteEngine on
RewriteMap blmap prg:/usr/local/bin/botdetect

RewriteCond %{REQUEST_URI} .html$
RewriteCond ${blmap:%{REMOTE_ADDR}|%{HTTP:X-FORWARDED-FOR}|%{REQUEST_URI}} =BLOCK
RewriteRule (.*) "-" [F]

In case you want to change the default parameters create a wrapper script and call botdetect from there with all the parameters you might want to set.

Usage

botdetect [options]

  -ignore-private-ips=true: igore private IPs when building the checksum
  -interval=5s: build a new blacklist after this much time
  -max-ratio=0.85: blacklist IPs if the app/assets ratio is above this threshold
  -max-requests=30: maximum number of requests to allow
  -timeout=10ms: wait this long for a redis response
  -timeslot=1m0s: the duration to use to group requests
  -timestamp-format="15:04": the key by which to group requests (golang time format, default: hour:minute)
  -trace=false: trace the decisions the program makes
  -version=false: Show the program version
  -window=1h0m0s: the time window to observe

Documentation

Index

• type Blacklist
• type History
  • func NewHistory(ctx context.Context, options *HistoryOptions) *History
  • func (h *History) Blacklist() Blacklist
  • func (h *History) IsBlacklisted(ip string) bool
  • func (h *History) RequestChannel() chan *Request
• type HistoryItem
• type HistoryOptions
• type IP
  • func NewIP() *IP
  • func (i *IP) IsPrivate(ip net.IP) bool
  • func (i *IP) Network(ip net.IP) *net.IPNet
• type Request

Types

type Blacklist

type Blacklist map[string]bool

Blacklist contains all blacklisted IP addresses as key

type History

type History struct {
	// contains filtered or unexported fields
}

History counts requests per IP for a given time window

func NewHistory

func NewHistory(ctx context.Context, options *HistoryOptions) *History

NewHistory creates a new History item

func (*History) Blacklist

func (h *History) Blacklist() Blacklist

Blacklist returns an array of IP addresses that are blacklisted

func (*History) IsBlacklisted

func (h *History) IsBlacklisted(ip string) bool

IsBlacklisted determines whether a given IP address is on the blacklist

func (*History) RequestChannel

func (h *History) RequestChannel() chan *Request

RequestChannel returns the channel through which IPs are fed to the history

type HistoryItem

type HistoryItem struct {
	Timestamp time.Time
	Count     uint64
	App       uint64
	Other     uint64
}

HistoryItem contains the request count for a given period of time denoted by Timestamp

type HistoryOptions

type HistoryOptions struct {
	TimestampFormat string
	TimeSlot        time.Duration
	Window          time.Duration
	Interval        time.Duration
	MaxRequests     uint64
	MaxRatio        float64
}

HistoryOptions configures the behaviour of History

type IP

type IP struct {
	IP net.IP
	// contains filtered or unexported fields
}

IP is a utility to check whether an IP address is private

func NewIP

func NewIP() *IP

NewIP creates a new IP structure

func (*IP) IsPrivate

func (i *IP) IsPrivate(ip net.IP) bool

IsPrivate checks whether a given IP address is privte

func (*IP) Network

func (i *IP) Network(ip net.IP) *net.IPNet

Network returns the private network in which the IP lies

type Request

type Request struct {
	URL string
	IP  net.IP
}

Request contains information the history needs about an HTTP request