go-dnscollector

README

DNS Collector

NOTE: The code before version 1.x is considered beta quality and is subject to breaking changes.

Overview

This tool acts as a high speed aggregator, analyzer, transporter and logging for your DNS logs, written in Golang. The DNS Traffic can be collected and aggregated simultaneously from many different sources (dnstap, sniffer, logs, etc.) .

This tool can also help you to visualize DNS traffic errors and anomalies with Grafana dashboard.

This tool can be used to transform dns queries or replies in JSON format with EDNS support.

{
 // query/reply ip and port, tcp/udp protocol and family used
 "network": {...},
 // message type, arrival packet time, latency.
 "dnstap": {...},
 // dns fields
 "dns": {...},
 // extended dns options like csubnet
 "edns": {...},
 // country, continent and city informations
 "geo": {...},
 // specific powerdns metadata like tags, original-request-subnet
 "pdns": {...}
}

Features

• Logs routing

• Queries/Replies JSON encoding

• Collectors:

  • DNStap streams
  • DNS packets sniffer
  • Tail on log file
  • Protobuf PowerDNS

• Loggers:

  • Console
  • Prometheus
  • File
  • DNStap
  • TCP
  • REST API
  • Syslog
  • Fluentd
  • Pcap
  • InfluxDB
  • Loki
  • Statsd
  • ElasticSearch

• Other features

  • DNS decoder with extended options support
  • Built-in Grafana dashboards
  • GeoIP support
  • Text format
  • DNS filtering
  • User Privacy
  • Normalize Qname

Installation

Run-it from binary

Download the binary from release page. If you want to integrate this tool with systemd, please to follow this guide.

./go-dnscollector -config config.yml

Run-it from dockerhub

Use the default config (dnstap -> stdout + rest api):

docker run -d --name=dnscollector01 dmachard/go-dnscollector

Override the default configuration (/etc/dnscollector/config.yml) with a config file on the host:

-v $(pwd)/config.yml:/etc/dnscollector/config.yml

Configuration

See the full Configuration guide for more details.

Use-cases

As prerequisites, we assume you have a DNS server which supports DNSTap (unbound, bind, powerdns, etc)

For more informations about dnstap, please to read the following page Dnstap: How to enable it on main dns servers

• Use case 1: Capture dns traffic (dnstap) and backup-it to text log files
• Use case 2: Capture dns traffic (dnstap) and get statistics usage with Prometheus + Grafana
• Use case 3: Convert captured dns traffic (dnstap) to JSON format
• Use case 4: Capture dns traffic (dnstap) and follow dns logs with Loki + Grafana
• Use case 5: Forward unix dnstap socket traffic to TLS dnstap stream
• Use case 6: Capture dns traffic with user privacy options enabled
• Use case 7: Running multiple dnstap collectors in parallel and aggregate logs in the same file
• Use case 8: Multiple PowerDNS collectors in parallel with prometheus metrics

Benchmark

Tested on the following machine: 8 vCPUs, 32 GB memory

packet per sec received	DnsCollector
50k	OK - 0% lost
100k	OK - 0% lost
150k	OK (0.07% lost)

For developers

Run from source

go run .

Execute testunits

go test -timeout 10s ./collectors/ -cover -v
go test -timeout 10s ./loggers/ -cover -v
go test -timeout 10s ./transformers/ -cover -v
go test -timeout 10s ./dnsutils/ -cover -v

Execute a test for one specific testcase in a package

go test -timeout 10s -cover -v ./loggers -run TestSyslogRunJsonMode

Building from source. Use the latest golang available on your target system

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o go-dnscollector *.go

Update package dependencies

go get github.com/dmachard/go-logger@v0.2.0
go get github.com/dmachard/go-powerdns-protobuf@v0.0.3
go get github.com/dmachard/go-dnstap-protobuf@v0.2.0
go mod tidy