README
¶
Bitfan
Data and Event processing pipeliner really inspired by Logstash.
Get bitfan, usage and configuration documentation and a availables processors
- Bitfan documentation https://bitfan.io
Features
- configuration file compatible with logstash config file format.
- support conditionals, env, sprintf variables in configuration : %{[field][key]} ${ENVVAR}
- supports input, filters, output and codecs
- consume local and remote (http) configuration files
- build complex pipelines with the
usekeyword to import, connect, fork to other pipelines/configuration files - gracefully stop and start each pipelines
- install bitfan as a system daemon / service
- manage running pipelines (list / stop / start a pipeline in a running bitfan)
- monitor pipeline processors and events with prometheus
- REST API to manage Bitfan
- WebUI
Similar projets in go
- tsaikd/gogstash - Logstash like, written in golang
- packetzoom/logzoom - A lightweight replacement for logstash indexer in Go
- hailocab/logslam - A lightweight lumberjack protocol compliant logstash indexer
- spartanlogs/spartan - Spartan is a data process much like Logstash
Credits
Icon made by Freepik from www.flaticon.com is licensed by CC 3.0 BY
Contributors
- @vjeantet - Valere JEANTET
- @mirdhyn - Merlin Gaillard
- @AlexAkulov - Alexander AKULOV
- @lor00x - Thomas GUILLIER
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
csv
Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec csv"; DO NOT EDIT
|
Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec csv"; DO NOT EDIT |
|
json
Code generated by "bitfanDoc -codec json"; DO NOT EDIT
|
Code generated by "bitfanDoc -codec json"; DO NOT EDIT |
|
jsonlines
Code generated by "bitfanDoc -codec json_lines"; DO NOT EDIT
|
Code generated by "bitfanDoc -codec json_lines"; DO NOT EDIT |
|
line
Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec
|
Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec |
|
multiline
The multiline codec will collapse multiline messages and merge them into a single event.
|
The multiline codec will collapse multiline messages and merge them into a single event. |
|
plain
Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec
|
Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec |
|
rubydebug
Code generated by "bitfanDoc -codec rubydebug"; DO NOT EDIT This codec pretty prints event
|
Code generated by "bitfanDoc -codec rubydebug"; DO NOT EDIT This codec pretty prints event |
|
w3c
Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec w3c"; DO NOT EDIT
|
Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec w3c"; DO NOT EDIT |
|
Entrypoint manage pipeline's definitions to get Pipeline ready to be used by the core
|
Entrypoint manage pipeline's definitions to get Pipeline ready to be used by the core |
|
filter-blacklist
The blacklist rule will check a certain field against a blacklist, and match if it is in the blacklist.
|
The blacklist rule will check a certain field against a blacklist, and match if it is in the blacklist. |
|
filter-change
This rule will monitor a certain field and match if that field changes.
|
This rule will monitor a certain field and match if that field changes. |
|
filter-date
The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.
|
The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event. |
|
filter-digest
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
filter-drop
Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received Drops everything that gets to this filter.
|
Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received Drops everything that gets to this filter. |
|
filter-eval
Code generated by "bitfanDoc "; DO NOT EDIT Modify or add event's field with the result of * an expression (math or compare) * a go template **Operators and types supported in expression :** * Modifiers: `+` `-` `/` `*` `&` `|` `^` `**` `%` `>>` `<<` * Comparators: `>` `>=` `<` `<=` `==` `!=` `=~` `!~` * Logical ops: `||` `&&` * Numeric constants, as 64-bit floating point (`12345.678`) * String constants (single quotes: `'foobar'`) * Date constants (single quotes, using any permutation of RFC3339, ISO8601, ruby date, or unix date; date parsing is automatically tried with any string constant) * Boolean constants: `true` `false` * Parenthesis to control order of evaluation `(` `)` * Arrays (anything separated by `,` within parenthesis: `(1, 2, 'foo')`) * Prefixes: `!` `-` `~` * Ternary conditional: `?` `:` * Null coalescence: `??`
|
Code generated by "bitfanDoc "; DO NOT EDIT Modify or add event's field with the result of * an expression (math or compare) * a go template **Operators and types supported in expression :** * Modifiers: `+` `-` `/` `*` `&` `|` `^` `**` `%` `>>` `<<` * Comparators: `>` `>=` `<` `<=` `==` `!=` `=~` `!~` * Logical ops: `||` `&&` * Numeric constants, as 64-bit floating point (`12345.678`) * String constants (single quotes: `'foobar'`) * Date constants (single quotes, using any permutation of RFC3339, ISO8601, ruby date, or unix date; date parsing is automatically tried with any string constant) * Boolean constants: `true` `false` * Parenthesis to control order of evaluation `(` `)` * Arrays (anything separated by `,` within parenthesis: `(1, 2, 'foo')`) * Prefixes: `!` `-` `~` * Ternary conditional: `?` `:` * Null coalescence: `??` |
|
filter-exec
Code generated by "bitfanDoc "; DO NOT EDIT Execute a command and use its stdout as event data
|
Code generated by "bitfanDoc "; DO NOT EDIT Execute a command and use its stdout as event data |
|
filter-geoip
Code generated by "bitfanDoc "; DO NOT EDIT The GeoIP filter adds information about the geographical location of IP addresses, based on data from the Maxmind GeoLite2 databases This processor use a GeoLite2 City database.
|
Code generated by "bitfanDoc "; DO NOT EDIT The GeoIP filter adds information about the geographical location of IP addresses, based on data from the Maxmind GeoLite2 databases This processor use a GeoLite2 City database. |
|
filter-grok
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
filter-html
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
filter-json
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
filter-kv
Code generated by "bitfanDoc "; DO NOT EDIT This filter helps automatically parse messages (or specific event fields) which are of the foo=bar variety.
|
Code generated by "bitfanDoc "; DO NOT EDIT This filter helps automatically parse messages (or specific event fields) which are of the foo=bar variety. |
|
filter-mutate
Code generated by "bitfanDoc "; DO NOT EDIT mutate filter allows to perform general mutations on fields.
|
Code generated by "bitfanDoc "; DO NOT EDIT mutate filter allows to perform general mutations on fields. |
|
filter-newterm
Code generated by "bitfanDoc "; DO NOT EDIT This processor matches when a new value appears in a field that has never been seen before.
|
Code generated by "bitfanDoc "; DO NOT EDIT This processor matches when a new value appears in a field that has never been seen before. |
|
filter-split
Code generated by "bitfanDoc "; DO NOT EDIT The split filter clones an event by splitting one of its fields and placing each value resulting from the split into a clone of the original event.
|
Code generated by "bitfanDoc "; DO NOT EDIT The split filter clones an event by splitting one of its fields and placing each value resulting from the split into a clone of the original event. |
|
filter-uuid
Code generated by "bitfanDoc "; DO NOT EDIT The uuid filter allows you to generate a UUID and add it as a field to each processed event.
|
Code generated by "bitfanDoc "; DO NOT EDIT The uuid filter allows you to generate a UUID and add it as a field to each processed event. |
|
filter-whitelist
Code generated by "bitfanDoc "; DO NOT EDIT Similar to blacklist, this processor will compare a certain field to a whitelist, and match if the list does not contain the term
|
Code generated by "bitfanDoc "; DO NOT EDIT Similar to blacklist, this processor will compare a certain field to a whitelist, and match if the list does not contain the term |
|
httpout
Code generated by "bitfanDoc "; DO NOT EDIT Display on http the last received event URL is available as http://webhookhost/pipelineName/pluginLabel/URI * webhookhost is defined by bitfan at startup * pluginLabel is defined in pipeline configuration, it's the named processor if you put one, or `httpout` by default * URI is defined in plugin configuration (see below)
|
Code generated by "bitfanDoc "; DO NOT EDIT Display on http the last received event URL is available as http://webhookhost/pipelineName/pluginLabel/URI * webhookhost is defined by bitfan at startup * pluginLabel is defined in pipeline configuration, it's the named processor if you put one, or `httpout` by default * URI is defined in plugin configuration (see below) |
|
httppoller
Code generated by "bitfanDoc "; DO NOT EDIT HTTPPoller allows you to intermittently poll remote HTTP URL, decode the output into an event
|
Code generated by "bitfanDoc "; DO NOT EDIT HTTPPoller allows you to intermittently poll remote HTTP URL, decode the output into an event |
|
input-beats
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-elasticsearch
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-event
Code generated by "bitfanDoc "; DO NOT EDIT Generate a blank event on interval
|
Code generated by "bitfanDoc "; DO NOT EDIT Generate a blank event on interval |
|
input-exec
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-file
Code generated by "bitfanDoc "; DO NOT EDIT Read file on * received event * when new file discovered this processor remember last files used, it stores references in sincedb, set it to "/dev/null" to not remember used files
|
Code generated by "bitfanDoc "; DO NOT EDIT Read file on * received event * when new file discovered this processor remember last files used, it stores references in sincedb, set it to "/dev/null" to not remember used files |
|
input-httpserver
Code generated by "bitfanDoc "; DO NOT EDIT Listen and read a http request to build events with it.
|
Code generated by "bitfanDoc "; DO NOT EDIT Listen and read a http request to build events with it. |
|
input-imap
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-rabbitmq
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-stdin
Code generated by "bitfanDoc "; DO NOT EDIT Read events from standard input.
|
Code generated by "bitfanDoc "; DO NOT EDIT Read events from standard input. |
|
input-stdout
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-syslog
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-tail
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-twitter
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-udp
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-unix
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
input-websocket
Code generated by "bitfanDoc "; DO NOT EDIT Receive event on a ws connection
|
Code generated by "bitfanDoc "; DO NOT EDIT Receive event on a ws connection |
|
ldap
Code generated by "bitfanDoc "; DO NOT EDIT Performs a search for a specified filter on the directory and fire events with results
|
Code generated by "bitfanDoc "; DO NOT EDIT Performs a search for a specified filter on the directory and fire events with results |
|
output-elasticsearch
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-elasticsearch2
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-email
Code generated by "bitfanDoc "; DO NOT EDIT Send email when an output is received.
|
Code generated by "bitfanDoc "; DO NOT EDIT Send email when an output is received. |
|
output-file
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-glusterfs
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-http
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-mongodb
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-null
Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received
|
Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received |
|
output-rabbitmq
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-statsd
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
output-tcp
Code generated by "bitfanDoc -processor output-tcp"; DO NOT EDIT
|
Code generated by "bitfanDoc -processor output-tcp"; DO NOT EDIT |
|
pop3
Code generated by "bitfanDoc "; DO NOT EDIT Periodically scan an POP3 mailbox for new emails.
|
Code generated by "bitfanDoc "; DO NOT EDIT Periodically scan an POP3 mailbox for new emails. |
|
route
Code generated by "bitfanDoc "; DO NOT EDIT This processor, depending on the condition evaluation, will route message to one or more different pipelines and/or pass the message through the processor to the next one.
|
Code generated by "bitfanDoc "; DO NOT EDIT This processor, depending on the condition evaluation, will route message to one or more different pipelines and/or pass the message through the processor to the next one. |
|
sleep
Code generated by "bitfanDoc "; DO NOT EDIT Sleep a given amount of time.
|
Code generated by "bitfanDoc "; DO NOT EDIT Sleep a given amount of time. |
|
sql
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
stdout
Code generated by "bitfanDoc "; DO NOT EDIT A simple output which prints to the STDOUT of the shell running BitFan.
|
Code generated by "bitfanDoc "; DO NOT EDIT A simple output which prints to the STDOUT of the shell running BitFan. |
|
stop
Code generated by "bitfanDoc "; DO NOT EDIT Stop after emitting a blank event on start Allow you to put first event and then stop processors as soon as they finish their job.
|
Code generated by "bitfanDoc "; DO NOT EDIT Stop after emitting a blank event on start Allow you to put first event and then stop processors as soon as they finish their job. |
|
template
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
|
use
Code generated by "bitfanDoc "; DO NOT EDIT When used in input (input->filter->o) the processor will receive events from the last filter from the pipeline used in configuration file.
|
Code generated by "bitfanDoc "; DO NOT EDIT When used in input (input->filter->o) the processor will receive events from the last filter from the pipeline used in configuration file. |
|
webfan
Code generated by "bitfanDoc "; DO NOT EDIT Example “` input{ webhook{ uri => "toto/titi" pipeline=> "test.conf" codec => plain{ role => "decoder" } codec => plain{ role => "encoder" format=> "<h1>Hello {{.request.querystring.name}}</h1>" } headers => { "Content-Type" => "text/html" } } } “`
|
Code generated by "bitfanDoc "; DO NOT EDIT Example “` input{ webhook{ uri => "toto/titi" pipeline=> "test.conf" codec => plain{ role => "decoder" } codec => plain{ role => "encoder" format=> "<h1>Hello {{.request.querystring.name}}</h1>" } headers => { "Content-Type" => "text/html" } } } “` |
|
websocket
Code generated by "bitfanDoc "; DO NOT EDIT Send event received over a ws connection to connected clients
|
Code generated by "bitfanDoc "; DO NOT EDIT Send event received over a ws connection to connected clients |
|
when
Code generated by "bitfanDoc "; DO NOT EDIT
|
Code generated by "bitfanDoc "; DO NOT EDIT |
Click to show internal directories.
Click to hide internal directories.