README
¶
Fluent Bit output plugins for CLP
Repository contains Fluent Bit output plugins that store records in CLP's compressed IR (intermediate representation) format. More details on IR can be found in this Uber Engineering Blog.
The general flow is as follows:
%%{init: {
'theme': 'base',
'themeVariables': {
'primaryColor': '#0066cc',
'primaryTextColor': '#fff',
'primaryBorderColor': 'transparent',
'lineColor': '#9580ff',
'secondaryColor': '#9580ff',
'tertiaryColor': '#fff'
}
}
}%%
flowchart LR
A(Fluent Bit Input) --> B
subgraph CLP Output Plugin
B(Parse into IR) --> C(Compress with Zstd)
end
C --> D(Output)
classDef format fill:#007DF4,color:white
class A,B,C,D format
Fluent Bit Input
Fluent Bit can collect application logs from >40 different sources. Common sources include tailing log files and other Fluent Bit instances.
CLP Output Plugin
Output plugin receives logs from Fluent Bit and parses them into CLP IR. CLP IR consists of a timestamp, a list of variable values, and the log type. IR is then compressed with Zstd in default mode without dictionaries.
Output
Compressed IR output is sent to plugin output (currently only AWS S3 is supported). CLP can directly ingest compressed IR output and convert into archives for efficient storage and search.
Usage
Each plugin has its own README to help get started. Currently, we only have a AWS S3 plugin, but please submit an issue if you need to send IR to another output.
Linting
- Install golangci-lint:
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | \
sh -s -- -b $(go env GOPATH)/bin v1.59.0
- Run with
golangci-lint run
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
decoder
Package implements Msgpack decoder.
|
Package implements Msgpack decoder. |
|
plugins
|
|
|
out_clp_s3
Package defines high-level callback functions required by Fluent Bit go plugin documentation.
|
Package defines high-level callback functions required by Fluent Bit go plugin documentation. |