mikrotik-hosts-parser

README

MikroTik hosts parser

This application provides HTTP server, that can generate script for RouterOS-based routers for blocking "AD" hosts using 3rd party host-lists (available by HTTP):

## Limit: 5000
## Cache lifetime: 30m0s
## Format: routeros
## Redirect to: 127.0.0.1
## Sources list:
##  - <https://adaway.org/hosts.txt>
##  - <https://www.malwaredomainlist.com/hostslist/hosts.txt>
## Excluded hosts:
##  - broadcasthost
##  - ip6-allhosts
##  - ip6-allnodes
##  - ip6-allrouters
## Cache HIT for <https://adaway.org/hosts.txt> (expires after 25m55s)
## Cache miss for <https://www.malwaredomainlist.com/hostslist/hosts.txt>

/ip dns static
add address=127.0.0.1 comment="ADBlock" disabled=no name="1-1ads.com"
add address=127.0.0.1 comment="ADBlock" disabled=no name="101com.com"
add address=127.0.0.1 comment="ADBlock" disabled=no name="101order.com"
add address=127.0.0.1 comment="ADBlock" disabled=no name="123freeavatars.com"

# ...

Hosts file format (example):

# Any comments
127.0.0.1   1-1ads.com
127.0.0.1   101com.com 101order.com
0.0.0.0     123freeavatars.com

All what you need is:

• Start current application HTTP server
• Make an HTTP request to the script generator endpoint /script/source?sources_urls=... with all required parameters (like records limit, hosts file URLs, exclusion list and others)
• Generated script source execute on your RouterOS-based hardware

More information can be found here.

Previous version (PHP) can be found in php-version branch.

Installing

Download latest binary file for your os/arch from releases page or use our docker image (ghcr.io). Also you may need in configuration file ./configs/config.yml and ./web directory content for web UI access.

Usage

This application supports next sub-commands:

Sub-command	Description
serve	Start HTTP server
healthcheck	Health checker for the HTTP server (use case - docker healthcheck) (hidden in CLI help)
version	Display application version

And global flags:

Flag	Description
--verbose, -v	Verbose output
--debug	Debug output
--log-json	Logs in JSON format

HTTP server starting

serve sub-command allows to use next flags:

Flag	Description	Default value	Environment variable
--listen, -l	IP address to listen on	0.0.0.0 (all interfaces)	LISTEN_ADDR
--port, -p	TCP port number	8080	LISTEN_PORT
--resources-dir, -r	Path to the directory with public assets	./web	RESOURCES_DIR
--config, -c	Config file path	./configs/config.yml	CONFIG_PATH
--caching-engine	Caching engine (memory or redis)	memory	CACHING_ENGINE
--cache-ttl	Cached entries lifetime (examples: 50s, 1h30m)	30m	CACHE_TTL
--redis-dsn	Redis server DSN, required only if redis caching engine is enabled	redis://127.0.0.1:6379/0	REDIS_DSN

Environment variables have higher priority then flag values.

Server starting command example:

$ ./mikrotik-hosts-parser serve \
    --config ./configs/config.yml \
    --listen 0.0.0.0 \
    --port 8080 \
    --resources-dir ./web

This command will start HTTP server using configuration from ./configs/config.yml on TCP port 8080 and use directory ./web for serving static files. Configuration file well-documented, so, feel free to change any settings on your choice!

Configuration file allows you to use environment variables with default values (used library)!

After that you can navigate your browser to http://127.0.0.1:8080/ and you will see something like that:

Special endpoint /script/source?sources_urls=... generates RouterOS-based script using passed http-get parameters (watch examples on index page).

Using docker

All supported image tags can be found here and here.

Just execute in your terminal:

$ docker run --rm -p 8080:8080/tcp tarampampam/mikrotik-hosts-parser:X.X.X

Where X.X.X is image tag (application version).

Demo

I can't guarantee that this links will available forever, but you can use this application by the following links:

• https://stopad.cgood.ru/

Testing

For application testing and building we use built-in golang testing feature and docker-ce + docker-compose as develop environment. So, just write into your terminal after repository cloning:

$ make test

Or build the binary file:

$ make build

Releasing

New versions publishing is very simple - just make required changes in this repository, update changelog file and "publish" new release using repo releases page.

Binary files and docker images will be build and published automatically.

New release will overwrite the latest docker image tag in both registers.

Changes log

Changes log can be found here.

Support

If you will find any package errors, please, make an issue in current repository.

License

This is open-sourced software licensed under the MIT License.