go-dnscollector

README

DNS logs aggregator and analyzer

Overview

This dns collector acts as a high speed aggregator and analyzer for your DNS logs, written in Golang.

Aggregate and route all your dns logs (queries and replies) from different sources (dnstap, sniffer, logs, etc.) to multiple destinations.

NOTE: The code before version 1.x is considered beta quality and is subject to breaking changes.

Features

• Logs routing

• Collectors:

  • DNStap streams
  • DNS packets sniffer
  • Tail on log file
  • Protobuf PowerDNS

• Transformers:

  • Queries/Replies JSON encoding
  • DNS filtering
  • User Privacy
  • Normalize Qname

• Loggers:

  • Stdout
  • File
  • DNStap
  • TCP
  • REST API
  • Syslog
  • Fluentd
  • Pcap
  • InfluxDB
  • Loki
  • Statsd

• Other features

  • DNS decoder with extended options support
  • Built-in Grafana dashboards
  • GeoIP support
  • Text format

Installation

Run-it from binary

Download the binary from release page. If you want to integrate this tool with systemd, please to follow this guide.

./go-dnscollector -config config.yml

Run-it from dockerhub

Use the default config (dnstap -> stdout + rest api):

docker run -d --name=dnscollector01 dmachard/go-dnscollector

Override the default configuration (/etc/dnscollector/config.yml) with a config file on the host:

-v $(pwd)/config.yml:/etc/dnscollector/config.yml

Configuration

See the full Configuration guide for more details.

Use-cases

As prerequisites, we assume you have a DNS server which supports DNSTap (unbound, bind, powerdns, etc)

For more informations about dnstap, please to read the following page Dnstap: How to enable it on main dns servers

• Use case 1: Capture dns traffic (dnstap) and backup-it to text log files
• Use case 2: Capture dns traffic (dnstap) and get statistics usage with Prometheus + Grafana
• Use case 3: Convert captured dns traffic (dnstap) to JSON format
• Use case 4: Capture dns traffic (dnstap) and follow dns logs with Loki + Grafana
• Use case 5: Forward unix dnstap socket traffic to TLS dnstap stream
• Use case 6: Capture dns traffic with user privacy options enabled
• Use case 7: Running multiple dnstap collectors in parallel

End to end testing

Tested with success with the following operating system and dns servers

Benchmark

Tested on the following machine: 8 vCPUs, 32 GB memory

packet per sec received	DnsCollector
50k	OK - 0% lost
100k	OK - 0% lost
150k	OK (0.07% lost)

For developers

Run from source

go run .

Execute testunits

go test -timeout 10s ./collectors/ -cover -v
go test -timeout 10s ./loggers/ -cover -v
go test -timeout 10s ./transformers/ -cover -v
go test -timeout 10s ./dnsutils/ -cover -v

Execute a test for one specific testcase in a package

go test -timeout 10s -cover -v ./loggers -run TestSyslogRunJsonMode

Building from source. Use the latest golang available on your target system

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o go-dnscollector *.go

Update package dependencies

go get github.com/dmachard/go-logger@v0.2.0
go get github.com/dmachard/go-powerdns-protobuf@v0.0.3
go get github.com/dmachard/go-dnstap-protobuf@v0.2.0
go mod tidy