Foretoken
A blazing fast, highly customizable, modern-day defence tool using (in memory) SQL & REST/gRPC protocols.
Report Bug
·
Request Feature
https://foretoken.domgolonka.com
Loved the project? Please consider donating to the bitcoin address below:
About
Foretoken is a tool to scrape and defend against potential dangerous threats faced on the internet. It aims to be a
highly customizable tools for companies and individuals to use to counter threats.
Features
- Emails (Rest/gRPC)
- Disposable
- Generic
- Free
- Spam
- IPs (Rest/gRPC)
- Score (Rest/gRPC)
- Database
- Editable Sources
Usage
Migrate
If using NON-MEMORY SQLITE or PostgreSQL, DO THIS BEFORE YOU RUN, You need to migrate the database:
make migrate
How to run
To run it on your local computer:
git clone https://github.com/domgolonka/foretoken
cd ./foretoken
make build && ./bin/foretoken
The default config file is config.yml
.
If you want to run it with a different config file (or add your own).
git clone https://github.com/domgolonka/foretoken
make build
(make sure to build it first)
./bin/foretoken --config=/PATH/TO/CONFIG
example:
./bin/foretoken --config=./config.prod.yml
Docker
You can run it in docker, locally:
docker build -t foretoken .
Once the image is built, Foretoken can be invoked by running the following:
docker run --rm -t -p 8080:8080 foretoken
Or run Docker from our repo:
docker run -d -p 8080:8080 domgolonka/foretoken
or with a custom config file:
docker run -d -p 8080:8080 domgolonka/foretoken --config=config.yml
Configs
All configurations are in the config files in the directory "config". You can add your own environment variables here.
External APIs
The application is improved if you sign up for external APIs. Leaked is paid, but all other services are free to sign up!
Change the file changeme.env
to .env
and save any External API Keys.
PWNEDKEY=
IP2LOCATION=
MAXMIND=
For full configuration examples, check out https://foretoken.domgolonka.com
Change the databases
At this moment, Foretoken only supports SQLite and PostgreSQL. You can change the databasename
field with
either postgresql
or sqlite3
By Default, the SQLite driver is set to "in memory". To use a file, you need to
specify that the host
to a .sqlite3
extension, example: YOURNAME.sqlite3
. This will create a
new SQLite file in the root directory.
For Postgresql, I would advise using a quick read/write database like timescale.
PostgreSQL is not yet tested
Score
The overall Fraud Score of the email and IP's reputation and recent behavior across the threat network. Fraud Scores >=
75 are suspicious, but not necessarily fraudulent.
This tool saves those threats on multiple different databases and uses REST API & gRPC to output data.
Source
All sources are available in the ./resource
directory. You can edit and the resources. They files get checked once a
day by the different modules.
Regular Expressions
Regex expressions are saved in the ./resource/expressions.json
file in JSON format.
Each regex looks like this:
{
"name": "ipv4",
"expression": "^((?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).*",
"type": "ipv4"
},
Name: The UNIQUE name of the regular expression.
Expression: The regex
Type: The type of expression. For IPs, it is
usually is a ipv4 or ipv6. For IP proxy, its http, https, sock4, sock5.
The files are stored in the ./resource
directory and start with ip_
such as ip_tor
for tor.
You can add sources by adding a new file to ./resource
directory and updating the config.yml
file:
### Resource files
resource:
emaildisposallist: [ "email_disposable" ]
emailfreelist: [ "email_free" ]
emailspamlist: [ "email_spam" ]
ipvpnlist: [ "ip_vpn" ]
ipopenvpnlist: [ "ip_openvpn" ]
iptorlist: [ "ip_tor" ]
ipproxylist: [ "ip_proxy" ]
ipspamlist: [ "ip_spam" ]
expressionlist: [ "expressions" ]
APIs
REST API & gRPC is enabled. For more API examples: https://foretoken.domgolonka.com
gRPC
The default gRPC port is 8082 (you can change in the config)
REST API
The REST API to the example app is described below.
Rate Limiting
You can enable the rate limiter for REST API in the config.yml
file.
ratelimit:
enabled: true
max: 20
expiration: 30
Max
number of recent connections during Duration
seconds before sending a 429 response
Expiration
is the time on how long to keep records of requests in memory per minute
Request
GET /health
curl -i -H 'Accept: application/json' http://localhost:8080/health
Response
HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:56:45 GMT
Content-Type: application/json
Content-Length: 13
{"http":true}
Request
GET /list/ip/proxy
curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/proxy
Response
HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 12 Feb 2021 03:21:38 GMT
Transfer-Encoding: chunked
{"result":[{"ID":1,"URL":"103.228.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.693099-05:00","UpdatedAt":"2020-12-04T19:12:05.693099-05:00","DeletedAt":null},{"ID":2,"URL":"196.3.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.69557-05:00","UpdatedAt":"2020-12-04T19:12:05.69557-05:00","DeletedAt":null},{"ID":3,"URL":"165.227.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.696224-05:00","UpdatedAt":"2020-12-04T19:12:05.696224-05:00","DeletedAt":null},{"ID":4,"URL":"117.197.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.696876-05:00","UpdatedAt":"2020-12-04T19:12:05.696876-05:00","DeletedAt":null},{"ID":5,"URL":"180.183.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.697515-05:00","UpdatedAt":"2020-12-04T19:12:05.697515-05:00","DeletedAt":null},{"ID":6,"URL":"159.192.xxx.xxx:8080","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.698074-05:00","UpdatedAt":"2020-12-04T19:12:05.698074-05:00","DeletedAt":null},{"ID":7,"URL":"185.28.xxx.xxx","Type":"ipv4","
Request
GET /list/ip/spam
curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/spam
Response
HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:57:33 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 34952
168.0.xxx.0/22
202.49.xxx.0/24
Request
GET /list/ip/vpn
curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/vpn
Response
HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
yul-c14.xxx.com
lim-c04.xxx.com
bhx-c05.xxx.com
Request
GET /list/ip/tor
curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/tor
Response
HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:58:18 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 23253
176.10.xxx.xxx
54.37.xxx.xxx
109.70.xxx.xxx
Request
GET /list/email/disposal
curl -i -H 'Accept: application/json' http://localhost:8080/list/email/disposal
Response
HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:58:18 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxx
xxx.cc
xxx.com
xxx.ca
Request
GET /list/email/generic
curl -i -H 'Accept: application/json' http://localhost:8080/list/email/generic
Response
HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:59:38 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxxx
xxx@
xxx@
xxx@
Request
GET /list/email/spam
curl -i -H 'Accept: application/json' http://localhost:8080/list/email/spam
Response
HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:59:38 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxxx
xxx.cc
xxx.com
xxx.ca
Request
GET /list/email/free
curl -i -H 'Accept: application/json' http://localhost:8080/list/email/free
Response
HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:59:38 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxxx
xxx.cc
xxx.com
xxx.ca
Request
GET /score/email/youremail@gmail.com
curl -i -H 'Accept: application/json' http://localhost:8080/score/email/youremail@gmail.com
Response
HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
10
Request
GET /score/ip/127.0.0.1
curl -i -H 'Accept: application/json' http://localhost:8080/score/ip/127.0.0.1
Response
HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
0
Request
GET /validate/email/youremail@gmail.com
curl -i -H 'Accept: application/json' http://localhost:8080/validate/email/youremail@gmail.com
Response
HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/json; charset=utf-8
Transfer-Encoding: chunked
{
"valid": true
}
Request
GET /email/youremail@gmail.com
curl -i -H 'Accept: application/json' http://localhost:8080/email/youremail@gmail.com
Response
HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/json; charset=utf-8
Transfer-Encoding: chunked
{
"valid": true,
"disposable": false,
"recent_spam": false,
"free": false,
"leaked": false,
"generic": false,
"score": 0,
"domain": {
"created_at": "1995-08-13T04:00:00Z",
"expiration_date": "2021-08-12T04:00:00Z"
}
}
Request
GET /ip/127.0.0.1
curl -i -H 'Accept: application/json' http://localhost:8080/ip/127.0.0.1
Response
HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/json; charset=utf-8
Transfer-Encoding: chunked
{
"success": false,
"proxy": false,
"ISP": "",
"organization": "",
"ASN": 0,
"host": "",
"country_code": "",
"city": "",
"region": "",
"is_crawler": false,
"connection_type": "",
"latitude": 0,
"longitude": 0,
"timezone": "",
"vpn": false,
"tor": false,
"recent_abuse": false,
"abuse_velocity": "",
"bot_status": false,
"mobile": false,
"score": 0,
"operating_system": "",
"browser": "",
"device_model": "",
"device_brand": ""
}
Service Discovery
Foretoken supports etcd3, zookeeper, and consul as a registry.
All service discovery configurations are stored in the config.yml
file:
servicediscovery:
service: ""
nodeid: ""
endpoint: ""
- Service: The viable options are
consul
, etc3
and zookeeper
- Nodeid: A name for the grpc nodeid
- endpoint: An address for the service such as zookeeper:
10.0.101.68:2189
, etcd: http://10.0.101.68:2379
or consul: http://10.0.101.68:8500
Metrics
Prometheus
Prometheus is enabled. Following metrices are available by default:
http_requests_total
http_request_duration_seconds
http_requests_in_progress_total
Work in progress
Lots of features are being worked on.
Roadmap
I would like a discussion going on the potential expansion of the tool.
I would like this tool to detect all modern threats.