Action plugins
add_file_name
It adds a field containing the file name to the event.
It is only applicable for input plugins k8s and file.
More details...
add_host
It adds field containing hostname to an event.
More details...
convert_date
It converts field date/time data to different format.
More details...
convert_log_level
It converts the log level field according RFC-5424.
More details...
convert_utf8_bytes
It converts multiple UTF-8-encoded bytes to corresponding characters.
Supports unicode (\u...
and \U...
), hex (\x...
) and octal (\{0-3}{0-7}{0-7}
) encoded bytes.
Note: Escaped and unescaped backslashes are treated the same.
For example, the following 2 values will be converted to the same result:
\x68\x65\x6C\x6C\x6F
and \\x68\\x65\\x6C\\x6C\\x6F
=> hello
Examples
pipelines:
example_pipeline:
...
actions:
- type: convert_utf8_bytes
fields:
- obj.field
...
The original event:
{
"obj": {
"field": "\\xD0\\xA1\\xD0\\x98\\xD0\\xA1\\xD0\\xA2\\xD0\\x95\\xD0\\x9C\\xD0\\x90.xml"
}
}
The resulting event:
{
"obj": {
"field": "СИСТЕМА.xml"
}
}
The original event:
{
"obj": {
"field": "$\\110\\145\\154\\154\\157\\054\\040\\146\\151\\154\\145\\056\\144!"
}
}
The resulting event:
{
"obj": {
"field": "$Hello, file.d!"
}
}
The original event:
{
"obj": {
"field": "$\\u0048\\u0065\\u006C\\u006C\\u006F\\u002C\\u0020\\ud801\\udc01!"
}
}
The resulting event:
{
"obj": {
"field": "$Hello, 𐐁!"
}
}
The original event:
{
"obj": {
"field": "{\"Dir\":\"C:\\\\Users\\\\username\\\\.prog\\\\120.67.0\\\\x86_64\\\\x64\",\"File\":\"$Storage$\\xD0\\x9F\\xD1\\x80\\xD0\\xB8\\xD0\\xB7\\xD0\\xBD\\xD0\\xB0\\xD0\\xBA.20.tbl.xml\"}"
}
}
The resulting event:
{
"obj": {
"field": "{\"Dir\":\"C:\\\\Users\\\\username\\\\.prog\\\\120.67.0\\\\x86_64\\\\x64\",\"File\":\"$Storage$Признак.20.tbl.xml\"}"
}
}
More details...
debug
It logs event to stderr. Useful for debugging.
It may sample by logging the first
N entries each tick.
If more events are seen during the same interval
,
every thereafter
message is logged and the rest are dropped.
For example,
- type: debug
interval: 1s
first: 10
thereafter: 5
This will log the first 10 events in a one second interval as-is.
Following that, it will allow through every 5th event in that interval.
More details...
decode
It decodes a string from the event field and merges the result with the event root.
If one of the decoded keys already exists in the event root, it will be overridden.
More details...
discard
It drops an event. It is used in a combination with match_fields
/match_mode
parameters to filter out the events.
An example for discarding informational and debug logs:
pipelines:
example_pipeline:
...
actions:
- type: discard
match_fields:
level: /info|debug/
...
More details...
flatten
It extracts the object keys and adds them into the root with some prefix. If the provided field isn't an object, an event will be skipped.
Example:
pipelines:
example_pipeline:
...
actions:
- type: flatten
field: animal
prefix: pet_
...
It transforms {"animal":{"type":"cat","paws":4}}
into {"pet_type":"b","pet_paws":"4"}
.
More details...
join
It makes one big event from the sequence of the events.
It is useful for assembling back together "exceptions" or "panics" if they were written line by line.
Also known as "multiline".
⚠ Parsing the whole event flow could be very CPU intensive because the plugin uses regular expressions.
Consider match_fields
parameter to process only particular events. Check out an example for details.
Example of joining Go panics:
pipelines:
example_pipeline:
...
actions:
- type: join
field: log
start: '/^(panic:)|(http: panic serving)/'
continue: '/(^\s*$)|(goroutine [0-9]+ \[)|(\([0-9]+x[0-9,a-f]+)|(\.go:[0-9]+ \+[0-9]x)|(\/.*\.go:[0-9]+)|(\(...\))|(main\.main\(\))|(created by .*\/.*\.)|(^\[signal)|(panic.+[0-9]x[0-9,a-f]+)|(panic:)/'
match_fields:
stream: stderr // apply only for events which was written to stderr to save CPU time
...
More details...
join_template
Alias to "join" plugin with predefined start
and continue
parameters.
⚠ Parsing the whole event flow could be very CPU intensive because the plugin uses regular expressions.
Consider match_fields
parameter to process only particular events. Check out an example for details.
Example of joining Go panics:
pipelines:
example_pipeline:
...
actions:
- type: join_template
template: go_panic
field: log
match_fields:
stream: stderr // apply only for events which was written to stderr to save CPU time
...
More details...
json_decode
It decodes a JSON string from the event field and merges the result with the event root.
If the decoded JSON isn't an object, the event will be skipped.
More details...
json_encode
It replaces field with its JSON string representation.
Example:
pipelines:
example_pipeline:
...
actions:
- type: json_encode
field: server
...
It transforms {"server":{"os":"linux","arch":"amd64"}}
into {"server":"{\"os\":\"linux\",\"arch\":\"amd64\"}"}
.
More details...
It extracts a field from JSON-encoded event field and adds extracted field to the event root.
If extracted field already exists in the event root, it will be overridden.
More details...
keep_fields
It keeps the list of the event fields and removes others.
More details...
mask
Mask plugin matches event with regular expression and substitutions successfully matched symbols via asterix symbol.
You could set regular expressions and submatch groups.
Example:
pipelines:
example_pipeline:
...
actions:
- type: mask
metric_subsystem_name: "some_name"
ignore_fields:
- trace_id
masks:
- re: "\b(\d{1,4})\D?(\d{1,4})\D?(\d{1,4})\D?(\d{1,4})\b"
groups: [1,2,3]
...
More details...
modify
It modifies the content for a field or add new field. It works only with strings.
You can provide an unlimited number of config parameters. Each parameter handled as cfg.FieldSelector
:cfg.Substitution
.
When _skip_empty
is set to true
, the field won't be modified/added in the case of field value is empty.
Note: When used to add new nested fields, each child field is added step by step, which can cause performance issues.
Example:
pipelines:
example_pipeline:
...
actions:
- type: modify
my_object.field.subfield: value is ${another_object.value}.
...
The resulting event could look like:
{
"my_object": {
"field": {
"subfield":"value is 666."
}
},
"another_object": {
"value": 666
}
Filters
Sometimes it is required to extract certain data from fields and for that purpose filter chains were added.
Filters are added one after another using pipe '|' symbol and they are applied to the last value in the chain.
For example, in expression ${field|re("(test-pod-\w+)",-1,[1],",")|re("test-pod-(\w+)",-1,[1],",")}
first the value of 'field' is retrieved,
then the data extracted using first regular expression and formed into a new string, then the second regular expression is applied
and its result is formed into a value to be put in modified field.
Currently available filters are:
-
regex filter
- re(regex string, limit int, groups []int, separator string[, emptyOnNotMatched bool])
, filters data using regex
, extracts limit
occurrences,
takes regex groups listed in groups
list, and if there are more than one extracted element concatenates result using separator
.
Negative value of limit
means all occurrences are extracted, limit
0 means no occurrences are extracted, limit
greater than 0 means
at most limit
occurrences are extracted.
Optional flag emptyOnNotMatched
allows to returns empty string if no matches occurred for regex.
-
trim filter
- trim(mode string, cutset string)
, trims data by the cutset
substring. Available modes are all
- trim both sides,
left
- trim only left, right
- trim only right.
Examples:
Example #1:
Data: {"message:"info: something happened"}
Substitution: level: ${message|re("(\w+):.*",-1,[1],",")}
Result: {"message:"info: something happened","level":"info"}
Example #2:
Data: {"message:"re1 re2 re3 re4"}
Substitution: extracted: ${message|re("(re\d+)",2,[1],",")}
Result: {"message:"re1 re2 re3 re4","extracted":"re1,re2"}
Example #3:
Data: {"message:"service=service-test-1 exec took 200ms"}
Substitution: took: ${message|re("service=([A-Za-z0-9_\-]+) exec took (\d+\.?\d*(?:ms|s|m|h))",-1,[2],",")}
Result: {"message:"service=service-test-1 exec took 200ms","took":"200ms"}
Example #4:
Data: {"message:"message without matching re"}
Substitution: extracted: ${message|re("test",1,[1],",",true)}
Result: {"message:"message without matching re","extracted":""}
Example #5:
Data: {"message:"{\"service\":\"service-test-1\",\"took\":\"200ms\"}\n"}
Substitution: message: ${message|trim("right","\n")}
Result: {"message:"{\"service\":\"service-test-1\",\"took\":\"200ms\"}"}
More details...
move
It moves fields to the target field in a certain mode.
In allow
mode, the specified fields
will be moved;
in block
mode, the unspecified fields
will be moved.
Examples
pipelines:
example_pipeline:
...
actions:
- type: move
mode: allow
target: other
fields:
- log.stream
- zone
...
The original event:
{
"service": "test",
"log": {
"level": "error",
"message": "error occurred",
"ts": "2023-10-30T13:35:33.638720813Z",
"stream": "stderr"
},
"zone": "z501"
}
The resulting event:
{
"service": "test",
"log": {
"level": "error",
"message": "error occurred",
"ts": "2023-10-30T13:35:33.638720813Z"
},
"other": {
"stream": "stderr",
"zone": "z501"
}
}
pipelines:
example_pipeline:
...
actions:
- type: move
mode: block
target: other
fields:
- log
...
The original event:
{
"service": "test",
"log": {
"level": "error",
"message": "error occurred",
"ts": "2023-10-30T13:35:33.638720813Z",
"stream": "stderr"
},
"zone": "z501",
"other": {
"user": "ivanivanov"
}
}
The resulting event:
{
"log": {
"level": "error",
"message": "error occurred",
"ts": "2023-10-30T13:35:33.638720813Z"
},
"other": {
"user": "ivanivanov",
"service": "test",
"zone": "z501"
}
}
More details...
parse_es
It parses HTTP input using Elasticsearch /_bulk
API format. It converts sources defining create/index actions to the events. Update/delete actions are ignored.
Check out the details in Elastic Bulk API.
More details...
parse_re2
It parses string from the event field using re2 expression with named subgroups and merges the result with the event root.
More details...
remove_fields
It removes the list of the event fields and keeps others.
Nested fields supported: list subfield names separated with dot.
Example:
fields: ["a.b.c"]
# event before processing
{
"a": {
"b": {
"c": 100,
"d": "some"
}
}
}
# event after processing
{
"a": {
"b": {
"d": "some" # "c" removed
}
}
}
If field name contains dots use backslash for escaping.
Example:
fields:
- exception\.type
# event before processing
{
"message": "Exception occurred",
"exception.type": "SomeType"
}
# event after processing
{
"message": "Exception occurred" # "exception.type" removed
}
More details...
rename
It renames the fields of the event. You can provide an unlimited number of config parameters. Each parameter handled as cfg.FieldSelector
:string
.
When override
is set to false
, the field won't be renamed in the case of field name collision.
Sequence of rename operations isn't guaranteed. Use different actions for prioritization.
Example:
pipelines:
example_pipeline:
...
actions:
- type: rename
override: false
my_object.field.subfield: new_sub_field
...
The resulting event could look like:
{
"my_object": {
"field": {
"new_sub_field":"value"
}
},
More details...
set_time
It adds time field to the event.
More details...
split
It splits array of objects into different events.
For example:
{
"data": [
{ "message": "go" },
{ "message": "rust" },
{ "message": "c++" }
]
}
Split produces:
{ "message": "go" },
{ "message": "rust" },
{ "message": "c++" }
Parent event will be discarded.
If the value of the JSON field is not an array of objects, then the event will be pass unchanged.
More details...
throttle
It discards the events if pipeline throughput gets higher than a configured threshold.
More details...
Generated using insane-doc