go-dnscollector

README

DNS-collector

NOTE: The code before version 1.x is considered beta quality and is subject to breaking changes.

DNS-collector acts as a passive high speed ingestor, aggregator and analyzer for your DNS traffic, written in Golang. The DNS traffic can be collected and aggregated from simultaneously sources like DNStap streams, network interface or log files and relays it to multiple other listeners with some transformations on it (traffic filtering, user privacy, ...) and DNS protocol conversions (to plain text, json, and more... ).

Additionally, DNS-collector also support

• Extension Mechanisms for DNS (EDNS) decoding
• IPv4/v6 defragmentation and TCP reassembly
• Nanoseconds in timestamps

Overview:

Collectors:

• Listen for logging traffic with streaming network protocols
  • DNStap with tls|tcp|unix transports support and proxifier
  • PowerDNS streams with full support
  • TZSP protocol support
• Live capture on a network interface
  • AF_PACKET socket with BPF filter
  • eBPF XDP ingress traffic
• Read text or binary files as input
  • Read and tail on Plain text files
  • Ingest PCAP or DNSTap files by watching a directory

Loggers:

• Local storage of your DNS logs in plain Text, Json, Pcap or Dnstap formats:
  • Stdout console with custom directives
  • File with automatic rotation and compression
• Provide metrics and API
  • Prometheus metrics and visualize-it with built-in dashboards for Grafana
  • Statsd support
  • REST API with swagger to search DNS domains
• Send to remote host with generic transport protocol
  • TCP
  • Syslog
  • DNSTap protobuf messages
• Send to various sinks
  • Fluentd
  • InfluxDB
  • Loki
  • ElasticSearch
  • Scalyr
  • Redis
  • Kafka
• Send to security tools
  • Falco

Transformers:

• Traffic Reducer
  • Detect repetitive queries/replies and log once
• Extractor
  • Add base64 encoded dns payload
• Latency Computing
  • Compute latency between replies and queries
  • Detect and count unanswered queries
• Traffic filtering
  • Downsampling
  • Dropping per Qname, QueryIP or Rcode
• User Privacy
  • Anonymize QueryIP
  • Minimaze Qname
  • Hash Query and Response IP with SHA1
• Normalize
  • Quiet Text
  • Qname to lowercase
  • Add TLD and TLD+1
• Geographical metadata
  • Country and City
• Suspicious traffic detector
  • Malformed and large packet
  • Uncommon Qtypes used
  • Unallowed chars in Qname
  • Excessive number of labels
  • Long Qname

Get Started

Download the latest release binary and start the DNS-collector with the provided configuration file. The default configuration listens on tcp/6000 for a DNSTap stream and DNS logs are printed on standard output.

./go-dnscollector -config config.yml

If you prefer run it from docker, follow this guide.

Configuration

The configuration of DNS-collector is done through a file named config.yml. When the DNS-collector starts, it will look for the config.yml from the current working directory.

See the full configuration guide for more details.

Examples:

You will find below some examples of configuration to manage your DNS logs.

• Capture DNS traffic from incoming DNSTap streams

  • Read from UNIX DNSTap socket and forward it to TLS stream
  • Transform DNSTap as input to JSON format as output
  • Relays DNSTap stream to multiple remote destination without decoding
  • Aggregate several DNSTap stream and forward it to the same file

• Capture DNS traffic from PowerDNS products

  • Capture multiple PowerDNS streams

• Observe your DNS traffic from logs

  • Observe DNS metrics with Prometheus and Grafana
  • Follow DNS traffic with Loki and Grafana

• Apply some transformations

  • Capture DNSTap stream and apply user privacy on it
  • Filtering incoming traffic with downsample and whitelist of domains
  • Transform all domains to lowercase
  • Add geographical metadata with GeoIP
  • Count the number of outgoing queries without replies

• Capture DNS traffic from FRSTRM/dnstap files

  • Save incoming DNStap streams to file (frstrm)
  • Watch for DNStap files as input

• Capture DNS traffic from PCAP files

  • Capture DNSTap stream and backup-it to text and pcap files
  • Watch for PCAP files as input and JSON as output

• Capture DNS traffic from Mikrotik device

  • Capture TZSP packets containing DNS packets and process them as json

Contributing

See the development guide for more information on how to build it yourself.

How to userguides:

• Add a new collector
• Add a new logger
• Add a new transform