bitfan

module
v0.0.0-...-691b052 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 7, 2018 License: Apache-2.0

README

Bitfan
Bitfan

Data and Event processing pipeliner really inspired by Logstash.

GoDoc Go Report Card Build Status Codecov

Get bitfan, usage and configuration documentation and a availables processors

Features

  • configuration file compatible with logstash config file format.
  • support conditionals, env, sprintf variables in configuration : %{[field][key]} ${ENVVAR}
  • supports input, filters, output and codecs
  • consume local and remote (http) configuration files
  • build complex pipelines with the use keyword to import, connect, fork to other pipelines/configuration files
  • gracefully stop and start each pipelines
  • install bitfan as a system daemon / service
  • manage running pipelines (list / stop / start a pipeline in a running bitfan)
  • monitor pipeline processors and events with prometheus
  • REST API to manage Bitfan
  • WebUI

Similar projets in go

  • tsaikd/gogstash - Logstash like, written in golang
  • packetzoom/logzoom - A lightweight replacement for logstash indexer in Go
  • hailocab/logslam - A lightweight lumberjack protocol compliant logstash indexer
  • spartanlogs/spartan - Spartan is a data process much like Logstash

Credits

Icon made by Freepik from www.flaticon.com is licensed by CC 3.0 BY

Contributors

  • @vjeantet - Valere JEANTET
  • @mirdhyn - Merlin Gaillard
  • @AlexAkulov - Alexander AKULOV
  • @lor00x - Thomas GUILLIER

Directories

Path Synopsis
api
client
models
cmd
bitfan
bitfan/commands
bitfanDoc
bitfanUI
bitfanUI/commands
bitfanUI/server
csv
Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec csv"; DO NOT EDIT
 Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec csv"; DO NOT EDIT
json
Code generated by "bitfanDoc -codec json"; DO NOT EDIT
 Code generated by "bitfanDoc -codec json"; DO NOT EDIT
jsonlines
Code generated by "bitfanDoc -codec json_lines"; DO NOT EDIT
 Code generated by "bitfanDoc -codec json_lines"; DO NOT EDIT
line
Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec
 Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec
multiline
The multiline codec will collapse multiline messages and merge them into a single event.
 The multiline codec will collapse multiline messages and merge them into a single event.
plain
Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec
 Code generated by "bitfanDoc -codec encoder"; DO NOT EDIT doc codec
rubydebug
Code generated by "bitfanDoc -codec rubydebug"; DO NOT EDIT This codec pretty prints event
 Code generated by "bitfanDoc -codec rubydebug"; DO NOT EDIT This codec pretty prints event
w3c
Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec w3c"; DO NOT EDIT
 Parses comma-separated value data into individual fields Code generated by "bitfanDoc -codec w3c"; DO NOT EDIT
xprocessor
memory
metrics
webhook
Entrypoint manage pipeline's definitions to get Pipeline ready to be used by the core
 Entrypoint manage pipeline's definitions to get Pipeline ready to be used by the core
parser
parser/logstash
doc
filter-blacklist
The blacklist rule will check a certain field against a blacklist, and match if it is in the blacklist.
 The blacklist rule will check a certain field against a blacklist, and match if it is in the blacklist.
filter-change
This rule will monitor a certain field and match if that field changes.
 This rule will monitor a certain field and match if that field changes.
filter-date
The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.
 The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.
filter-digest
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
filter-drop
Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received Drops everything that gets to this filter.
 Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received Drops everything that gets to this filter.
filter-eval
Code generated by "bitfanDoc "; DO NOT EDIT Modify or add event's field with the result of * an expression (math or compare) * a go template **Operators and types supported in expression :** * Modifiers: `+` `-` `/` `*` `&` `|` `^` `**` `%` `>>` `<<` * Comparators: `>` `>=` `<` `<=` `==` `!=` `=~` `!~` * Logical ops: `||` `&&` * Numeric constants, as 64-bit floating point (`12345.678`) * String constants (single quotes: `'foobar'`) * Date constants (single quotes, using any permutation of RFC3339, ISO8601, ruby date, or unix date; date parsing is automatically tried with any string constant) * Boolean constants: `true` `false` * Parenthesis to control order of evaluation `(` `)` * Arrays (anything separated by `,` within parenthesis: `(1, 2, 'foo')`) * Prefixes: `!` `-` `~` * Ternary conditional: `?` `:` * Null coalescence: `??`
 Code generated by "bitfanDoc "; DO NOT EDIT Modify or add event's field with the result of * an expression (math or compare) * a go template **Operators and types supported in expression :** * Modifiers: `+` `-` `/` `*` `&` `|` `^` `**` `%` `>>` `<<` * Comparators: `>` `>=` `<` `<=` `==` `!=` `=~` `!~` * Logical ops: `||` `&&` * Numeric constants, as 64-bit floating point (`12345.678`) * String constants (single quotes: `'foobar'`) * Date constants (single quotes, using any permutation of RFC3339, ISO8601, ruby date, or unix date; date parsing is automatically tried with any string constant) * Boolean constants: `true` `false` * Parenthesis to control order of evaluation `(` `)` * Arrays (anything separated by `,` within parenthesis: `(1, 2, 'foo')`) * Prefixes: `!` `-` `~` * Ternary conditional: `?` `:` * Null coalescence: `??`
filter-exec
Code generated by "bitfanDoc "; DO NOT EDIT Execute a command and use its stdout as event data
 Code generated by "bitfanDoc "; DO NOT EDIT Execute a command and use its stdout as event data
filter-geoip
Code generated by "bitfanDoc "; DO NOT EDIT The GeoIP filter adds information about the geographical location of IP addresses, based on data from the Maxmind GeoLite2 databases This processor use a GeoLite2 City database.
 Code generated by "bitfanDoc "; DO NOT EDIT The GeoIP filter adds information about the geographical location of IP addresses, based on data from the Maxmind GeoLite2 databases This processor use a GeoLite2 City database.
filter-grok
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
filter-html
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
filter-json
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
filter-kv
Code generated by "bitfanDoc "; DO NOT EDIT This filter helps automatically parse messages (or specific event fields) which are of the foo=bar variety.
 Code generated by "bitfanDoc "; DO NOT EDIT This filter helps automatically parse messages (or specific event fields) which are of the foo=bar variety.
filter-mutate
Code generated by "bitfanDoc "; DO NOT EDIT mutate filter allows to perform general mutations on fields.
 Code generated by "bitfanDoc "; DO NOT EDIT mutate filter allows to perform general mutations on fields.
filter-newterm
Code generated by "bitfanDoc "; DO NOT EDIT This processor matches when a new value appears in a field that has never been seen before.
 Code generated by "bitfanDoc "; DO NOT EDIT This processor matches when a new value appears in a field that has never been seen before.
filter-split
Code generated by "bitfanDoc "; DO NOT EDIT The split filter clones an event by splitting one of its fields and placing each value resulting from the split into a clone of the original event.
 Code generated by "bitfanDoc "; DO NOT EDIT The split filter clones an event by splitting one of its fields and placing each value resulting from the split into a clone of the original event.
filter-uuid
Code generated by "bitfanDoc "; DO NOT EDIT The uuid filter allows you to generate a UUID and add it as a field to each processed event.
 Code generated by "bitfanDoc "; DO NOT EDIT The uuid filter allows you to generate a UUID and add it as a field to each processed event.
filter-whitelist
Code generated by "bitfanDoc "; DO NOT EDIT Similar to blacklist, this processor will compare a certain field to a whitelist, and match if the list does not contain the term
 Code generated by "bitfanDoc "; DO NOT EDIT Similar to blacklist, this processor will compare a certain field to a whitelist, and match if the list does not contain the term
httpout
Code generated by "bitfanDoc "; DO NOT EDIT Display on http the last received event URL is available as http://webhookhost/pipelineName/pluginLabel/URI * webhookhost is defined by bitfan at startup * pluginLabel is defined in pipeline configuration, it's the named processor if you put one, or `httpout` by default * URI is defined in plugin configuration (see below)
 Code generated by "bitfanDoc "; DO NOT EDIT Display on http the last received event URL is available as http://webhookhost/pipelineName/pluginLabel/URI * webhookhost is defined by bitfan at startup * pluginLabel is defined in pipeline configuration, it's the named processor if you put one, or `httpout` by default * URI is defined in plugin configuration (see below)
httppoller
Code generated by "bitfanDoc "; DO NOT EDIT HTTPPoller allows you to intermittently poll remote HTTP URL, decode the output into an event
 Code generated by "bitfanDoc "; DO NOT EDIT HTTPPoller allows you to intermittently poll remote HTTP URL, decode the output into an event
input-beats
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-elasticsearch
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-event
Code generated by "bitfanDoc "; DO NOT EDIT Generate a blank event on interval
 Code generated by "bitfanDoc "; DO NOT EDIT Generate a blank event on interval
input-exec
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-file
Code generated by "bitfanDoc "; DO NOT EDIT Read file on * received event * when new file discovered this processor remember last files used, it stores references in sincedb, set it to "/dev/null" to not remember used files
 Code generated by "bitfanDoc "; DO NOT EDIT Read file on * received event * when new file discovered this processor remember last files used, it stores references in sincedb, set it to "/dev/null" to not remember used files
input-httpserver
Code generated by "bitfanDoc "; DO NOT EDIT Listen and read a http request to build events with it.
 Code generated by "bitfanDoc "; DO NOT EDIT Listen and read a http request to build events with it.
input-imap
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-rabbitmq
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-stdin
Code generated by "bitfanDoc "; DO NOT EDIT Read events from standard input.
 Code generated by "bitfanDoc "; DO NOT EDIT Read events from standard input.
input-stdout
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-syslog
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-tail
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-twitter
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-udp
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-unix
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
input-websocket
Code generated by "bitfanDoc "; DO NOT EDIT Receive event on a ws connection
 Code generated by "bitfanDoc "; DO NOT EDIT Receive event on a ws connection
ldap
Code generated by "bitfanDoc "; DO NOT EDIT Performs a search for a specified filter on the directory and fire events with results
 Code generated by "bitfanDoc "; DO NOT EDIT Performs a search for a specified filter on the directory and fire events with results
output-elasticsearch
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-elasticsearch2
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-email
Code generated by "bitfanDoc "; DO NOT EDIT Send email when an output is received.
 Code generated by "bitfanDoc "; DO NOT EDIT Send email when an output is received.
output-file
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-glusterfs
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-http
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-mongodb
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-null
Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received
 Code generated by "bitfanDoc "; DO NOT EDIT Drops everything received
output-rabbitmq
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-statsd
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
output-tcp
Code generated by "bitfanDoc -processor output-tcp"; DO NOT EDIT
 Code generated by "bitfanDoc -processor output-tcp"; DO NOT EDIT
pop3
Code generated by "bitfanDoc "; DO NOT EDIT Periodically scan an POP3 mailbox for new emails.
 Code generated by "bitfanDoc "; DO NOT EDIT Periodically scan an POP3 mailbox for new emails.
route
Code generated by "bitfanDoc "; DO NOT EDIT This processor, depending on the condition evaluation, will route message to one or more different pipelines and/or pass the message through the processor to the next one.
 Code generated by "bitfanDoc "; DO NOT EDIT This processor, depending on the condition evaluation, will route message to one or more different pipelines and/or pass the message through the processor to the next one.
sleep
Code generated by "bitfanDoc "; DO NOT EDIT Sleep a given amount of time.
 Code generated by "bitfanDoc "; DO NOT EDIT Sleep a given amount of time.
sql
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
stdout
Code generated by "bitfanDoc "; DO NOT EDIT A simple output which prints to the STDOUT of the shell running BitFan.
 Code generated by "bitfanDoc "; DO NOT EDIT A simple output which prints to the STDOUT of the shell running BitFan.
stop
Code generated by "bitfanDoc "; DO NOT EDIT Stop after emitting a blank event on start Allow you to put first event and then stop processors as soon as they finish their job.
 Code generated by "bitfanDoc "; DO NOT EDIT Stop after emitting a blank event on start Allow you to put first event and then stop processors as soon as they finish their job.
template
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
testutils
use
Code generated by "bitfanDoc "; DO NOT EDIT When used in input (input->filter->o) the processor will receive events from the last filter from the pipeline used in configuration file.
 Code generated by "bitfanDoc "; DO NOT EDIT When used in input (input->filter->o) the processor will receive events from the last filter from the pipeline used in configuration file.
webfan
Code generated by "bitfanDoc "; DO NOT EDIT Example “` input{ webhook{ uri => "toto/titi" pipeline=> "test.conf" codec => plain{ role => "decoder" } codec => plain{ role => "encoder" format=> "<h1>Hello {{.request.querystring.name}}</h1>" } headers => { "Content-Type" => "text/html" } } } “`
 Code generated by "bitfanDoc "; DO NOT EDIT Example “` input{ webhook{ uri => "toto/titi" pipeline=> "test.conf" codec => plain{ role => "decoder" } codec => plain{ role => "encoder" format=> "<h1>Hello {{.request.querystring.name}}</h1>" } headers => { "Content-Type" => "text/html" } } } “`
websocket
Code generated by "bitfanDoc "; DO NOT EDIT Send event received over a ws connection to connected clients
 Code generated by "bitfanDoc "; DO NOT EDIT Send event received over a ws connection to connected clients
when
Code generated by "bitfanDoc "; DO NOT EDIT
 Code generated by "bitfanDoc "; DO NOT EDIT
xprocessor

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.