botex

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 17, 2021 License: AGPL-3.0 Imports: 45 Imported by: 0

README

Botex

Botex is an open source bad bot management tool from ScraperWall. Botex enables you to protect your website from automated attacks on your application logic such as content and price scraping, account takeovers, mysterious load spikes or skewed metrics.

It can be integrated into Apache, Nginx, Varnish or third party WAFs such as Cloudflare or simply be used to analyze your website traffic in real time as a first step to evaluate whether integrating it into the web stack completely makes sense.

Installation

If you've got the Go toolchain installed, botex can easily be installed by go

$ go get github.com/scraperwall/botex/cmd/botex

Binary releases are available for Linux, OS X and Windows.

Analyze a log file

If you only want to check if your website has a bad bot problem you can replay a single log file. For this to work, botex needs to know the format of the log file and its location.

$ botex \
    -log-replay /var/log/nginx/access.log.0 \
    -log-format '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'

botex uses nginx variable names. It needs $remote_addr, $time_local and $request as a bare minimum to analyze a log file. Make sure you use these names even if you analyze a log file from a different server such as Apache.

botex loads the log file so that its entire content fits into the time frame you've set it to observe.

Time Frame

botex looks at requests in a time window you can set. An hour is a good starting point.

$ botex -window-size 1m -num-windows 60

makes botex use 60 slices, each one minute long. This means that incoming HTTP requests get grouped into one-minute blocks and expire after 60 minutes.

Detection configuration

The open source version of botex performs a heuristic analysis of incoming HTTP requests. It calculates the ratio of HTML URLs to those of assets (images, CSS, Javascript) on an IP, network and autonomous system level.

If an IP exceeds the specified ratio towards HTML URLs and a configurable number of total requests it gets flagged as bad.

$ botex -min-app-requests 10 -max-app-requests 150 -max-ratio 0.91

tells botex to not flag any IP as bad if has less than 11 requests. Once it has made more requests in the specified time frame it will get flagged as bad only if the ratio of total requests : HTML requests is bigger than 0.91, i.e. the client has downloaded much more HTML content than assets.

This approach works well even if you use a CDN for assets since not all images will be cached in the CDN all the time.

Network and Autonomous System analysis

botex will perform the heuristic analysis on entire networks and autonomous systems. VPNs and proxy farms usually rent IPs from a limited number of providers in a limited number of networks. Suppose an attacker rents 200 IPs from a proxy farm. This allows him to perform a much larger number of requests before botex detects him but since his IPs come from a small number of networks and an even smaller number of autonomous systems (i.e. companies) looking at the total : HTML ratio based on networks and autonomous systems detects him easily and makes it much more cumbersome for him to bypass the detection.

$ botex -networks
DNS Lookups

botex doesn't flag any IP before it hasn't looked up its host name successfully. The DNS lookup process happens in real time and resolved IPs are being cached. Depending on how many requests per second your web server receives you can make botex use a corresponding number of resolver threads.

$ botex -resolver-workers 25 -dns-server 192.168.1.1:53

makes botex use 25 parallel DNS resolver workers that all use 192.168.1.1 as name server. If the number of requests is too much for your own name server, you may use a free public name server, e.g. from Google or Cloudflare. 25 threads is a good starting point for small to medium websites.

GeoIP and ASN databases

For its analysis botex requires two free databases from Maxmind: "GeoLite2 ASN: CSV Format" (ZIP) and "GeoLite2 City" (MMDB, gzip).

At the time of writing you had to sign up for an account to download the databases.

$ botex \
    -asndb-file ./GeoLite2-ASN-CSV_20210511.zip \
    -geoipdb-file ./GeoLite2-City_20210511.tar.gz
Request logging

If you would like to see which URLs an IP has requested, botex remembers the latest requests.

$ botex -keep-requests 100

lets botex keep the 100 latest requests per IP address.

Whitelisting

There are guaranteed to be IP addresses which you don't want blocked no matter what. Those IPs can be whitelisted by a TOML file using a wide range of filters.

[[ClientHost]]
Pattern = ".+\\.googlebot\\.com"
Description = "Googlebot"

[[ClientHost]]
Pattern = ".+\\.search\\.msn\\.com"
Description = "Bing Bot"

[[ASN]]
Pattern = "714"
Description = "Apple"

[[IP]]
Pattern = "165\\.73\\.12\\.251"
Description = "Monitoring"

[[CIDR]]
Pattern = "23.15.155.0/24"
Description = "Partner Network"

[[ServerHost]]
Pattern = "cdn.+\\.mydomain\\.com"
Description = "CDN"

[[ServerPath]]
Pattern = "/checkout/.+"
Description = "Checkout Process"

[[Useragent]]
Pattern = "mymonitor/1\\..+ (136228b8b06f9296)"
Description = "Internal Monitor"

All Patterns except for ASN and CIDR are evaluated as regular expressions. Backslashes (\) need to be escaped as demonstrated above.

ClientHost can be used to whitelist remote hosts such as those from which the Google Bot visits websites.

With ASN an entire autonomous system (i.e. an entire company) can be whitelisted. In the example above Apple is whitelisted through its autonomous system number (ASN) 714.

IP can whitelist IP addresses. Since the pattern is evaluated as a regular expression whitelisting more than one IP is possible:

Pattern = "112\\.47\\..+

whitelistes all IPs that start with 112.47.

CIDR can be used to whitelist entire networks. Use the CIDR notation as demonstrated above.

If there are hosts on your side that you don't want to be observed for some reason, use ServerHost to whitelist them.

Pattern = "cdn.+\\.mysite\\.com"

would whitelist all traffic to your CDN servers, e.g. cdn1.mysite.com or cdn-34-euwest.mysite.com.

ServerPath can in analogy be used to ignore traffic for specific URLs, for example if you don't want botex to count requests once a customer has entered the checkout process.

With Useragent you can whitelist clients by their user agent. This should only be the last resort if all other whitelist options fail since faking a user agent is extremely easy.

Complete Example

Suppose your log file contains log entries that are formatted as follows

94.134.88.168 - - [31/Mar/2020:10:00:00 +0200] "GET / HTTP/1.1" 200 12573 "https://scraperwall.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36"

the corresponding log file format specification for botex is

'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'

and in order to analyze the log file you would start botex with the following parameters:

$ botex \
   -log-replay=/var/log/nginx/access.log.0 \
   -log-format '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$none"' \
       -resolver-workers 50 \
       -dns-server 192.168.1.1:53 \
       -asndb-file ./GeoLite2-ASN-CSV_20210511.zip \
       -geoipdb-file ./GeoLite2-City_20210511.tar.gz \
       -window-size 1m \
       -keep-requests 500 \
       -num-windows 60 \
       -max-app-requests 150 \
       -max-ratio 0.91  \
       -networks \
	   -whitelist ./whitelist.toml \
       -clear-blocked

-clear-blocked makes botex clear all blocked IPs from the database when it starts.

API

The list of blocked IPs can be retrieved via an API in real-time. The API listens on port 4343.

$ curl -s http://127.0.0.1:4343/blocked/ips
 
[{
    "total": 134,
    "app": 134,
    "other": 0,
    "ratio": 1,
    "ips": 0,
    "asn": {
      "network": {
        "IP": "3.120.0.0",
        "Mask": "//gAAA=="
      },
      "from": "3.120.0.0",
      "to": "3.127.255.255",
      "cidr": "3.120.0.0/13",
      "asn": 16509,
      "organization": "AMAZON-02"
    },
    "reason": "too many requests (13/10) and app/asset ratio too high (1.00/0.91)",
    "city": {
      "name": "Frankfurt am Main",
      "continent": "Europe",
      "continent_code": "EU",
      "country": "Germany",
      "country_code": "DE",
      "accuracy_radius": 1000,
      "latitude": 50.1188,
      "longitude": 8.6843,
      "metro_code": 0,
      "timezone": "Europe/Berlin",
      "postcode": "60313",
      "registered_country": "United States",
      "registered_country_code": "US",
      "represented_country": "",
      "represented_country_code": "",
      "represented_country_type": "",
      "subdivisions": [
        "Hesse"
      ],
      "is_anonymous_proxy": false,
      "is_satellite_provider": false
    },
    "blocked_at": "2021-05-12T15:51:29.288571+02:00",
    "ip": "3.123.36.206",
    "hostname": "ec2-3-123-36-206.eu-central-1.compute.amazonaws.com"
}]
Everything about a single IP address

All available information about a single IP address can be retrieved, too:

$ curl -s | jq .
{
  "ip_details": {
    "ip": "107.178.98.165",
    "hostname": "we.love.servers.at.ioflood.net",
    "asn": {
      "network": {
        "IP": "107.178.64.0",
        "Mask": "///AAA=="
      },
      "from": "107.178.64.0",
      "to": "107.178.127.255",
      "cidr": "107.178.64.0/18",
      "asn": 53755,
      "organization": "IOFLOOD"
    },
    "geoip": {
      "ip": "107.178.98.165",
      "anonymous": {
        "is_anonymous": false,
        "is_anonymous_vpn": false,
        "is_hosting_provider": false,
        "is_public_proxy": false,
        "is_tor_exit_node": false
      },
      "city": {
        "name": "Phoenix",
        "continent": "North America",
        "continent_code": "NA",
        "country": "United States",
        "country_code": "US",
        "accuracy_radius": 1000,
        "latitude": 33.4413,
        "longitude": -112.0421,
        "metro_code": 753,
        "timezone": "America/Phoenix",
        "postcode": "85034",
        "registered_country": "United States",
        "registered_country_code": "US",
        "represented_country": "",
        "represented_country_code": "",
        "represented_country_type": "",
        "subdivisions": [
          "Arizona"
        ],
        "is_anonymous_proxy": false,
        "is_satellite_provider": false
      },
      "country": {
        "continent_code": "NA",
        "continent": "North America",
        "country_code": "US",
        "country": "United States",
        "registered_country_code": "US",
        "registered_country": "United States",
        "represented_country_code": "",
        "represented_country_type": "",
        "represented_country": "",
        "is_anonymous_proxy": false,
        "is_satellite_provider": false
      },
      "domain": {
        "domain": ""
      },
      "isp": {
        "autonomous_system_number": 0,
        "autonomous_system_organization": "",
        "isp": "",
        "organization": ""
      }
    },
    "total": 1,
    "app": 1,
    "other": 0,
    "ratio": 1,
    "is_blocked": true,
    "block_reason": "asn has too many requests (2797/150) and ratio is too high (1.00/0.91)",
    "whitelisted": false,
    "whitelist_reason": "",
    "created_at": "2021-05-12T16:20:21.360999+02:00",
    "updated_at": "2021-05-12T16:21:45.991959+02:00",
    "lastblock_at": "2021-05-12T16:22:21.428609+02:00"
  },
  "requests": [
    {
      "url": "/axess/Axess-2020.html",
      "host": "scw.test",
      "useragent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36",
      "source": "107.178.98.165",
      "method": "GET",
      "seq": 0,
      "timestamp": 1620829305991955000,
      "time": "2021-05-12T16:21:45.991955+02:00",
      "asn": {
        "network": {
          "IP": "107.178.64.0",
          "Mask": "///AAA=="
        },
        "from": "107.178.64.0",
        "to": "107.178.127.255",
        "cidr": "107.178.64.0/18",
        "asn": 53755,
        "organization": "IOFLOOD"
      },
      "is_app": true
    }
  ],
  "useragents": {
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36": 1
  }
}

Web GUI

There is a web GUI available at github.com/scraperwall/botex-admin

License

Unless otherwise noted, the botex source files are distributed under the GNU AGPLv3 license found in the LICENSE file.

Documentation

Index

Constants

View Source 
const DNSLookupError = "lookup error"

DNSLookupError denotes a dns lookup error

Variables

This section is empty.

Functions

This section is empty.

Types

type API 

type API struct {
	// contains filtered or unexported fields
}

API provides the HTTP REST API for botex

func NewAPI 

func NewAPI(ctx context.Context, config *config.Config, botex *Botex) (api *API, err error)

NewAPI creates a new REST-API for botex

type Block 

type Block struct {
	// contains filtered or unexported fields
}

Block is used to add IPs to the blocklist and to check whether an IP is blocked

func NewBlock 

func NewBlock(ctx context.Context, wlChan, blChan chan bool, resources *Resources, blockTTL time.Duration) *Block

NewBlock creates a new Blocklist. The parent context and application configuration are passed on to the new instance

func (*Block) ASNNamespace 

func (b *Block) ASNNamespace(asn int) []byte

func (*Block) AllASNs 

func (b *Block) AllASNs() ([]asndb.ASN, error)

AllASNs returns all currently blocked IPs

func (*Block) AllIPs 

func (b *Block) AllIPs() ([]*data.IPBlockMessage, error)

AllIPs returns all currently blocked IPs

func (*Block) AllNetworks 

func (b *Block) AllNetworks() ([]net.IPNet, error)

AllNetworks returns all currently blocked IPs

func (*Block) BlockASN 

func (b *Block) BlockASN(msg data.BlockMessage) error

BlockaASN writes an entire autonomous system (AS) to the block list It returns an error if writing the information failed

func (*Block) BlockIP 

func (b *Block) BlockIP(msg data.IPBlockMessage) error

BlockIP writes an IPs details to the blocklist. It returns an error if writing the information failed

func (*Block) BlockNetwork 

func (b *Block) BlockNetwork(msg data.NetworkBlockMessage) error

BlockNetwork writes an entire network to the block list It returns an error if writing the information failed

func (*Block) BlockedASNs 

func (b *Block) BlockedASNs() []data.BlockMessage

func (*Block) BlockedIPs 

func (b *Block) BlockedIPs() []data.IPBlockMessage

func (*Block) BlockedNetworks 

func (b *Block) BlockedNetworks() []data.NetworkBlockMessage

func (*Block) CIDRNamespace 

func (b *Block) CIDRNamespace(cidr string) []byte

func (*Block) CheckBlocked 

func (b *Block) CheckBlocked()

func (*Block) Clear 

func (b *Block) Clear()

Clear removes all currently blocked items from the store

func (*Block) CountASNs 

func (b *Block) CountASNs() int

CountASNs returns the number of currently blocked networks

func (*Block) CountIPs 

func (b *Block) CountIPs() int

CountIPs returns the number of currently blocked IPs

func (*Block) CountNetworks 

func (b *Block) CountNetworks() int

CountNetworks returns the number of currently blocked networks

func (*Block) GetASN 

func (b *Block) GetASN(asn *asndb.ASN) (*data.BlockMessage, error)

GetASN retrieves a blocked ASN. If the ASN isn't blocked an error is returned

func (*Block) GetIP 

func (b *Block) GetIP(ip net.IP) (*IPDetails, error)

GetIP retrieves an IPDetails item about a blocked IP. If the IP isn't blocked an error is returned

func (*Block) GetNetwork 

func (b *Block) GetNetwork(network net.IPNet) (*data.BlockMessage, error)

GetNetworkretrieves an IPDetails item about a blocked IP. If the IP isn't blocked an error is returned

func (*Block) IPNamespace 

func (b *Block) IPNamespace(ip []byte) []byte

func (*Block) IsBlocked 

func (b *Block) IsBlocked(ip net.IP, asn *asndb.ASN) bool

func (*Block) IsBlockedByASN 

func (b *Block) IsBlockedByASN(asn *asndb.ASN) (blocked bool, reason string)

func (*Block) RemoveASN 

func (b *Block) RemoveASN(asn *asndb.ASN) error

RemoveNetwork removes a network from the blocklist

func (*Block) RemoveIP 

func (b *Block) RemoveIP(ip net.IP) error

RemoveIP removes an IP from the blocklist

func (*Block) RemoveNetwork 

func (b *Block) RemoveNetwork(network net.IPNet) error

RemoveNetwork removes a network from the blocklist

type Botex 

type Botex struct {
	// contains filtered or unexported fields
}

Botex detects bad bots

func New 

func New(ctx context.Context, config *config.Config) (*Botex, error)

New creates a new Botex instance

func (*Botex) HandleRequest 

func (b *Botex) HandleRequest(r *data.Request)

HandleRequest handles incoming requests

func (*Botex) LogReplay 

func (b *Botex) LogReplay(logfile, format string, anonymize bool)

LogReplay replays a log file into the application

func (*Botex) Use 

func (b *Botex) Use(p Plugin)

Use adds a plugin to botex

type CIDRWhitelistRule 

type CIDRWhitelistRule struct {
	Network     *net.IPNet
	Description string
}

CIDRWhitelistRule is a CIDR whitelist rule

type Decision 

type Decision int
const (
	IsBlocked Decision = iota
	IsHuman
	IsWhitelisted
)

type History 

type History struct {
	// contains filtered or unexported fields
}

History is the history of all IPs for which the application has received a request

func NewHistory 

func NewHistory(ctx context.Context, plugins []Plugin, resources *Resources, config *config.Config) *History

NewHistory creates a new History item and passes on the context and configuration from its parent

func (*History) Add 

func (h *History) Add(r *data.Request) (ipd *IPData, newIP bool)

Add adds a single HTTP request to the history If Add has added a new item to the data map it returns true, otherwise it returns false

func (*History) Each 

func (h *History) Each(callback func(key string, ipd *IPData))

func (*History) IPData 

func (h *History) IPData(ip net.IP) *IPData

IPData returns the IPData struct for the given IP

func (*History) IPDetails 

func (h *History) IPDetails(ip net.IP) *IPDetails

IPDetails returns the IPDetails for the given IP

func (*History) SetHostname 

func (h *History) SetHostname(ip net.IP, hostname string)

SetHostname sets the reverse hotname for a given IP

func (*History) Size 

func (h *History) Size() int

Size returns the number of IPs in the history

func (*History) TotalStats 

func (h *History) TotalStats() data.IPStats

TotalStats returns the sum of the stats for all IPs

type IPData 

type IPData struct {
	IPDetails `json:"ipdetails"`
	Requests  *Requests `json:"requests"`
	// contains filtered or unexported fields
}

IPData contains IPDetails and the most recent HTTP requests. It handles updating the aggregated stats when it receives new requests

func NewIPData 

func NewIPData(updateChan chan data.IPStats, ip net.IP, plugins []Plugin, resources *Resources, config *config.Config) *IPData

NewIPData creates a new IPData item fro a given IP. the parent context and app configuration are passed on from the parent

func (*IPData) Add 

func (ipd *IPData) Add(r *data.Request) Decision

Add adds a single HTTP request

func (*IPData) Expire 

func (ipd *IPData) Expire() int

Expire removes all expired requests

func (*IPData) IPStats 

func (ipd *IPData) IPStats() data.IPStats

func (*IPData) SetHostname 

func (ipd *IPData) SetHostname(hostname string)

SetHostname sets the reverse hostname for an IP

func (*IPData) ShouldBeBlocked 

func (ipd *IPData) ShouldBeBlocked() bool

ShouldBeBlocked deterines whether the IP represented by this item should be blocked

func (*IPData) Stats 

func (ipd *IPData) Stats() data.Stats

func (*IPData) Update 

func (ipd *IPData) Update(stats data.IPStats, force bool)

Update sets the cached statistics numbers using an IPStats item

type IPDetails 

type IPDetails struct {
	IP              net.IP       `json:"ip"`
	Hostname        string       `json:"hostname"`
	ASN             *asndb.ASN   `json:"asn"`
	GeoIP           *geoip.GeoIP `json:"geoip"`
	Total           int          `json:"total"`
	App             int          `json:"app"`
	Other           int          `json:"other"`
	Ratio           float64      `json:"ratio"`
	IsBlocked       bool         `json:"is_blocked"`
	BlockReason     string       `json:"block_reason"`
	Whitelisted     bool         `json:"whitelisted"`
	WhitelistReason string       `json:"whitelist_reason"`
	CreatedAt       time.Time    `json:"created_at"`
	UpdatedAt       time.Time    `json:"updated_at"`
	LastBlockAt     time.Time    `json:"lastblock_at"`
	ForceBlock      bool         `json:"-"`
}

IPDetails contains meta information about an IP, its aggregated statistics and a reason for why it was blocked

type IPResolv 

type IPResolv struct {
	IP     net.IP    `json:"ip"`
	Host   string    `json:"host"`
	Err    string    `json:"err"`
	Tries  int       `json:"tries"`
	TEnd   time.Time `json:"tend"`
	TStart time.Time `json:"tstart"`
}

IPResolv contains an IP address and its corresponding reverse hostname

func NewIPResolv 

func NewIPResolv(ip net.IP) *IPResolv

NewIPResolv creates a new IP that needs to be resolved

func (*IPResolv) TimeTaken 

func (rip *IPResolv) TimeTaken() time.Duration

TimeTaken returns the amount of time it has taken to resolve the IP

type MapWindow 

type MapWindow struct {
	// contains filtered or unexported fields
}

MapWindow is a map that contains a rolling.TimePolicy for each entry

func NewMapWindow 

func NewMapWindow(windowSize time.Duration, numWindows int) *MapWindow

NewMapWindow creates a new MapWindow. windowSize determines the size of each window, numWindows how many windows there should be The context of the parent gets passed on to the new instance

func (*MapWindow) Add 

func (mw *MapWindow) Add(req *data.Request)

Add adds one item for the given key

func (*MapWindow) Expire 

func (mw *MapWindow) Expire() int

Expire removes expired items

func (*MapWindow) Size 

func (mw *MapWindow) Size() int

Size returns how many entries the MapWindow has got

func (*MapWindow) Total 

func (mw *MapWindow) Total() int

Total determines the sum of all map entries

func (*MapWindow) TotalMap 

func (mw *MapWindow) TotalMap() map[string]int

TotalMap returns a map that contains the key along with the sum of its entries

type Plugin 

type Plugin interface {
	HandleRequest(r *data.Request)
	APIHooks(r *gin.Engine)
	SetBlocker(b data.Blocker)
	ShouldBeBlocked(stats data.IPStats) (blocked bool, reason string)
	IsWhitelisted(ip net.IP) bool
}

type Requests 

type Requests struct {
	IP net.IP
	// contains filtered or unexported fields
}

Requests contains all HTTP requests for a given time. The requests are divied by their type: app and other (assets) Requests notifies its parent of changes through updateChan

func NewRequests 

func NewRequests(ip net.IP, updateChan chan data.IPStats, config *config.Config) *Requests

NewRequests creates a new Requests item. The app context and configuration get passed into the new item

func (*Requests) Add 

func (r *Requests) Add(req *data.Request)

Add adds a request

func (*Requests) App 

func (r *Requests) App() int

App returns the number of all application requests

func (*Requests) AppStats 

func (r *Requests) AppStats() map[int]int64

AppStats returns a map that has the timestamp (unix nanoseconds) as key and the count of app requests as value

func (*Requests) CanBeExpired 

func (r *Requests) CanBeExpired() bool

CanBeExpired determines whether there are requests that can be expired

func (*Requests) Expire 

func (r *Requests) Expire() int

Expire expires everything and updates the stats

func (*Requests) Latest 

func (r *Requests) Latest() []*data.Request

Latest returns the most recent requests

func (*Requests) Other 

func (r *Requests) Other() int

Other returns the number of all non-app requests

func (*Requests) Ratio 

func (r *Requests) Ratio() float64

Ratio returns the ratio of app requests / total requests

func (*Requests) Total 

func (r *Requests) Total() int

Total returns the total number of requests

func (*Requests) TotalStats 

func (r *Requests) TotalStats() map[int]int64

TotalStats returns a map that has the timestamp (unix nanoseconds) as key and the count of requests as value

func (*Requests) Useragents 

func (r *Requests) Useragents() map[string]int

Useragents returns a map made up of the user agent and its responding count

type RequestsWindow 

type RequestsWindow struct {
	// contains filtered or unexported fields
}

RequestsWindow contains the most recent requests The number of requests is limited by a maximum number of requests the list may contain (maxSize) and ttl, the time requests stay in the list before they expire and are removed

func NewRequestsWindow 

func NewRequestsWindow(maxRequests int, windowSize time.Duration, numWindows int) *RequestsWindow

NewRequestsWindow creates a new RequestsWindow. The app context and configuration get passed into the new item

func (*RequestsWindow) Add 

func (rw *RequestsWindow) Add(r *data.Request)

Add adds a single request

func (*RequestsWindow) Expire 

func (rw *RequestsWindow) Expire() int

Expire removes expired requests from the window

func (*RequestsWindow) Len 

func (rw *RequestsWindow) Len() int

Len returns the number of items in a RequestsWindow

func (*RequestsWindow) Requests 

func (rw *RequestsWindow) Requests() []*data.Request

Requests returns an array of all requests in the list

type Resolver 

type Resolver struct {
	// contains filtered or unexported fields
}

Resolver is a DNS resolver with Redis cache

func NewResolver 

func NewResolver(ctx context.Context, resources *Resources, config *config.Config) (*Resolver, error)

NewResolver creates a new Resolver item

func (*Resolver) Resolve 

func (r *Resolver) Resolve(rip *IPResolv)

Resolve queues a ReverseResolvable to be resolved

func (*Resolver) StartWorkers 

func (r *Resolver) StartWorkers(outChan chan *IPResolv) error

StartWorkers starts the resolver workers. It pulls IPs from the input queue and processes them internally. when an IP has been resolved the result is sent over the output channel

type Resources 

type Resources struct {
	ASNDB         *asndb.DB
	GEOIPDB       *geoip.DB
	NatsServer    *natsd.Server
	NatsConn      *nats.Conn
	Store         store.KVStore
	Whitelist     *Whitelist
	Resolver      *Resolver
	WebsocketChan chan interface{}

	BlockChan chan *IPDetails
}

func NewResources 

func NewResources() *Resources

type Stats 

type Stats struct {
	Total       int64     `json:"total"`
	Whitelisted int64     `json:"whitelisted"`
	Blocked     int64     `json:"blocked"`
	Human       int64     `json:"human"`
	Time        time.Time `json:"time,string"`
	UpdatedAt   time.Time `json:"updated_at,string"`
}

func (Stats) MarshalJSON 

func (s Stats) MarshalJSON() ([]byte, error)

func (*Stats) UnmarshalJSON 

func (s *Stats) UnmarshalJSON(bytes []byte) error

type StatsWindows 

type StatsWindows struct {
	Stats
	Map *treemap.Map `json:"data"`
	// contains filtered or unexported fields
}

func NewStatsWindows 

func NewStatsWindows(resources *Resources, config *config.Config) *StatsWindows

func (*StatsWindows) Add 

func (s *StatsWindows) Add(d Decision, t time.Time) error

func (*StatsWindows) All 

func (s *StatsWindows) All() map[string]Stats

func (*StatsWindows) Expire 

func (s *StatsWindows) Expire()

type WebserverSocket 

type WebserverSocket struct {
	// contains filtered or unexported fields
}

func NewWebserverSocket 

func NewWebserverSocket(ctx context.Context, config *config.Config, block *Block, requestHandler func(*data.Request)) (*WebserverSocket, error)

type Whitelist 

type Whitelist struct {
	UpdatedAt time.Time
	// contains filtered or unexported fields
}

Whitelist is the structure that contains all whitelist logic

func NewWhitelist 

func NewWhitelist(ctx context.Context, blocklistRecheckChan chan bool, config *config.Config) (*Whitelist, error)

NewWhitelist creates a new whitelist data structure from MongoDB

func (*Whitelist) IsCIDRWhitelisted 

func (wl *Whitelist) IsCIDRWhitelisted(ip net.IP) (bool, string)

IsCIDRWhitelisted determines if the given IP is part of a network specified by the cidr

func (*Whitelist) IsURLWhitelisted 

func (wl *Whitelist) IsURLWhitelisted(url string) bool

IsURLWhitelisted determines whether a requested URL is whitelisted and should not be processed

func (*Whitelist) IsWhitelisted 

func (wl *Whitelist) IsWhitelisted(ipd *IPDetails) (whitelisted bool, description string)

IsWhitelisted determines whether the IP represented by ipd is whitelisted. The method returns whether the IP is whitelisted and the description of the rule that matched

func (*Whitelist) IsWhitelistedByServerHost 

func (wl *Whitelist) IsWhitelistedByServerHost(r *data.Request) bool

IsWhitelistedByServerHost checks whether an incoming request is whitelisted by a server hostname rule

func (*Whitelist) IsWhitelistedByServerPath 

func (wl *Whitelist) IsWhitelistedByServerPath(r *data.Request) bool

IsWhitelistedByServerPath checks whether an incoming request is whitelisted by a server path (URL) rule

func (*Whitelist) IsWhitelistedByUseragent 

func (wl *Whitelist) IsWhitelistedByUseragent(r *data.Request) bool

IsWhitelistedByUseragent checks whether an incoming request is whitelisted by a useragent rule

func (*Whitelist) Load 

func (wl *Whitelist) Load() error

Load retrieves all whitelist rules from the configuration filee

type WhitelistRegexps 

type WhitelistRegexps struct {
	IP        *regexp.Regexp
	Host      *regexp.Regexp
	Org       *regexp.Regexp
	ASN       *regexp.Regexp
	Target    *regexp.Regexp
	URL       *regexp.Regexp
	UserAgent *regexp.Regexp
}

WhitelistRegexps contains whitelist regexps for various data fields

type WhitelistRule 

type WhitelistRule struct {
	Pattern     string
	Description string
	Regexp      *regexp.Regexp
}

WhitelistRule represents a single Whitelist rule

type WhitelistRules 

type WhitelistRules struct {
	IP         []WhitelistRule
	CIDR       []WhitelistRule
	ClientHost []WhitelistRule
	Org        []WhitelistRule
	ASN        []WhitelistRule
	Useragent  []WhitelistRule
	ServerHost []WhitelistRule
	ServerPath []WhitelistRule
}

WhitelistRules contains the Whitelist configuration

type Window 

type Window struct {
	// contains filtered or unexported fields
}

Window implements a rolling window using a TreeMap as storage

func NewWindow 

func NewWindow(windowSize time.Duration, numWindows int) *Window

NewWindow creates a new TreemapWindow instance with numWindows buckets that each cover a windowSize time range TODO: remove ctx as arg

func (*Window) Add 

func (w *Window) Add(t time.Time)

Add adds an item to the Treemap instance

func (*Window) Count 

func (w *Window) Count() int64

Count returns the total count of items in all buckets

func (*Window) Expire 

func (w *Window) Expire() int

Expire removes expired items from the window

func (*Window) Map 

func (w *Window) Map() map[int]int64

Source Files

View all Source files

Directories

Path Synopsis
cmd
badger
botex

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.