README
¶
Flow Pipeline
About The Project
bwNET is a research project of the German federal state of Baden-Württemberg which aims to provide innovative services within the state's research and education network BelWü. This GitHub Org contains the code pertaining to the monitoring aspect of this project.
This repo contains our flow processing toolkit which enables us and our users to define pipelines for goflow2-compatible flow messages. The flowpipeline project integrates most other parts of our flow processing stack into a single piece of software which can be configured to serve any function:
- accepting raw Netflow (using goflow2)
- enriching the resulting flow messages (examples/enricher)
- writing to and reading from Kafka (examples/localkafka)
- dumping flows to cli (e.g. flowdump)
- providing metrics and insights (examples/prometheus)
- and many more...
Getting Started
To get going, choose one of the following deployment methods.
Compile from Source
Clone this repo and use go build . to build the binary yourself.
By default, the binary will look for a config.yml in its local directory, so
you'll either want to create one or call it from any example directory (and
maybe follow the instructions there).
Binary Releases
Download our latest release and run it, same as if you compiled it yourself.
Container Releases
A ready to use container is provided as bwnetflow/flowpipeline, you can check
it out on docker hub.
Configurations referencing other files (geolocation databases for instance)
will work in a container without extra edits. This is because the volume
mountpoint /config is prepended in all segments which accept configuration to
open files, if the binary was built with the container build flag.
podman run -v ./examples/xy:/config flowpipeline
# or
docker run -v ./examples/xy:/config flowpipeline
Configuration
Refer to CONFIGURATION.md
for the full guide. Other than that, looking at the examples should give you a
good idea what the config looks like in detail and what the possible
applications are. For sake of completeness, here's another minimal example
which starts listening for Netflow v9 on port 2055, applies the filter given as
first argument, and then prints it out in a tcpdump-style format.
- segment: goflow
- segment: flowfilter
config:
filter: $0
- segment: printflowdump
You'd call it with ./flowpipeline "proto tcp and (port 80 or port 443)"., for
instance.
Contributing
Contributions in any form (code, issues, feature requests) are very much welcome.
Documentation
¶
Overview ¶
The flowpipeline utility unifies all bwNetFlow functionality and provides configurable pipelines to process flows in any manner.
The main entrypoint accepts command line flags to point to a configuration file and to establish the log level.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
The pipeline package manages segments in Pipeline objects.
|
The pipeline package manages segments in Pipeline objects. |
|
This package is home to all pipeline segment implementations.
|
This package is home to all pipeline segment implementations. |
|
alert/http
This segment is used to alert on flows using webhooks - WIP, but basically usable.
|
This segment is used to alert on flows using webhooks - WIP, but basically usable. |
|
export/influx
Collects and exports all flows to influxdb for long term storage.
|
Collects and exports all flows to influxdb for long term storage. |
|
export/prometheus
Collects and serves statistics about flows.
|
Collects and serves statistics about flows. |
|
filter/elephant
Filters out the bulky average of flows.
|
Filters out the bulky average of flows. |
|
filter/flowfilter
Runs flows through a filter and forwards only matching flows.
|
Runs flows through a filter and forwards only matching flows. |
|
input/goflow
Captures Netflow v9 and feeds flows to the following segments.
|
Captures Netflow v9 and feeds flows to the following segments. |
|
input/kafkaconsumer
Consumes flows from a Kafka instance and passes them to the following segments.
|
Consumes flows from a Kafka instance and passes them to the following segments. |
|
input/stdin
Receives flows from stdin in JSON format, as exported by the json segment.
|
Receives flows from stdin in JSON format, as exported by the json segment. |
|
modify/addcid
Enriches any passing flow message with a customer id field based on a CIDR match.
|
Enriches any passing flow message with a customer id field based on a CIDR match. |
|
modify/anonymize
Anonymize uses the CryptoPan prefix-preserving IP address sanitization as specified by J. Fan, J. Xu, M. Ammar, and S. Moon.
|
Anonymize uses the CryptoPan prefix-preserving IP address sanitization as specified by J. Fan, J. Xu, M. Ammar, and S. Moon. |
|
modify/dropfields
Drops fields from any passing flow.
|
Drops fields from any passing flow. |
|
modify/geolocation
Enriches flows with a geolocation.
|
Enriches flows with a geolocation. |
|
modify/normalize
Rewrites passing flows with all sampling rate affected fields normalized.
|
Rewrites passing flows with all sampling rate affected fields normalized. |
|
modify/protomap
Enriches any passing flow message with the human readable protocol.
|
Enriches any passing flow message with the human readable protocol. |
|
modify/remoteaddress
Determines the remote address of flows based on different criteria.
|
Determines the remote address of flows based on different criteria. |
|
modify/snmp
Enriches passing flows with human-readable versions of interface ids, sourced from SNMP.
|
Enriches passing flows with human-readable versions of interface ids, sourced from SNMP. |
|
noop
Serves as a template for new segments and forwards flows, otherwise does nothing.
|
Serves as a template for new segments and forwards flows, otherwise does nothing. |
|
output/csv
Package csv processes all flows from it's In channel and converts them into CSV format.
|
Package csv processes all flows from it's In channel and converts them into CSV format. |
|
output/json
Prints all flows to stdout or a given file in json format, for consumption by the stdin segment or for debugging.
|
Prints all flows to stdout or a given file in json format, for consumption by the stdin segment or for debugging. |
|
output/kafkaproducer
Produces all received flows to Kafka instance.
|
Produces all received flows to Kafka instance. |
|
output/sqlite
Dumps all incoming flow messages to a local sqlite database.
|
Dumps all incoming flow messages to a local sqlite database. |
|
print/count
Counts the number of passing flows and prints the result on termination.
|
Counts the number of passing flows and prints the result on termination. |
|
print/printdots
Prints a dot every n flows.
|
Prints a dot every n flows. |
|
print/printflowdump
Prints all incoming flows in a specific flowdump format.
|
Prints all incoming flows in a specific flowdump format. |