decode

package
v0.50.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 21, 2025 License: BSD-3-Clause Imports: 7 Imported by: 0

README

Decode plugin

It decodes a string from the event field and merges the result with the event root.

If one of the decoded keys already exists in the event root, it will be overridden.

Examples

JSON decoder

JSON decoder is used by default, so there is no need to specify it explicitly. You can specify json_max_fields_size in params to limit the length of string fields.

If the decoded JSON isn't an object, the event will be skipped.

Default decoder:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      prefix: p_
    ...

The original event:

{
  "level": "error",
  "log": "{\"message\":\"error occurred\",\"ts\":\"2023-10-30T13:35:33.638720813Z\",\"stream\":\"stderr\"}",
  "service": "test"
}

The resulting event:

{
  "level": "error",
  "service": "test",
  "p_message": "error occurred",
  "p_ts": "2023-10-30T13:35:33.638720813Z",
  "p_stream": "stderr"
}

Decoder with json_max_fields_size param:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: json
      params:
        json_max_fields_size:
          message: 5
          ts: 10
    ...

The original event:

{
  "level": "error",
  "log": "{\"message\":\"error occurred\",\"ts\":\"2023-10-30T13:35:33.638720813Z\",\"stream\":\"stderr\"}",
  "service": "test"
}

The resulting event:

{
  "level": "error",
  "service": "test",
  "message": "error",
  "ts": "2023-10-30",
  "stream": "stderr"
}
Postgres decoder
pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: postgres
    ...

The original event:

{
  "level": "error",
  "log": "2021-06-22 16:24:27 GMT [7291] => [3-1] client=test_client,db=test_db,user=test_user LOG:  listening on Unix socket \"/var/run/postgresql/.s.PGSQL.5432\"",
  "service": "test"
}

The resulting event:

{
  "level": "error",
  "service": "test",
  "time": "2021-06-22 16:24:27 GMT",
  "pid": "7291",
  "pid_message_number": "3-1",
  "client": "test_client",
  "db": "test_db",
  "user": "test_user",
  "log": "listening on Unix socket \"/var/run/postgresql/.s.PGSQL.5432\""
}
Nginx-error decoder

The event root may contain any of the following fields:

  • time string
  • level string
  • pid string
  • tid string
  • cid string
  • message string

You can specify nginx_with_custom_fields: true in params to decode custom fields.

Default decoder:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: nginx_error
    ...

The original event:

{
  "level": "warn",
  "log": "2022/08/17 10:49:27 [error] 2725122#2725122: *792412315 lua udp socket read timed out, context: ngx.timer",
  "service": "test"
}

The resulting event:

{
  "service": "test",
  "time": "2022/08/17 10:49:27",
  "level": "error",
  "pid": "2725122",
  "tid": "2725122",
  "cid": "792412315",
  "message": "lua udp socket read timed out, context: ngx.timer"
}

Decoder with nginx_with_custom_fields param:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: nginx_error
      params:
        nginx_with_custom_fields: true
    ...

The original event:

{
  "level": "warn",
  "log": "2022/08/18 09:29:37 [error] 844935#844935: *44934601 upstream timed out (110: Operation timed out), while connecting to upstream, client: 10.125.172.251, server: , request: \"POST /download HTTP/1.1\", upstream: \"http://10.117.246.15:84/download\", host: \"mpm-youtube-downloader-38.name.tldn:84\"",
  "service": "test"
}

The resulting event:

{
  "service": "test",
  "time": "2022/08/18 09:29:37",
  "level": "error",
  "pid": "844935",
  "tid": "844935",
  "cid": "44934601",
  "message": "upstream timed out (110: Operation timed out), while connecting to upstream",
  "client": "10.125.172.251",
  "server": "",
  "request": "POST /download HTTP/1.1",
  "upstream": "http://10.117.246.15:84/download",
  "host": "mpm-youtube-downloader-38.name.tldn:84"
}
Protobuf decoder

For correct decoding, the protocol scheme and message name are required. They must be specified in params.

Decoder with proto-file path:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: protobuf
      params:
        proto_file: 'path/to/proto/example.proto'
        proto_message: MyMessage
    ...

Decoder with proto-file content:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: protobuf
      params:
        proto_file: |
          syntax = "proto3";

          package example;
          option go_package = "example.v1";

          message Data {
            string string_data = 1;
            int32 int_data = 2;
          }

          message MyMessage {
            message InternalData {
              repeated string my_strings = 1;
              bool is_valid = 2;
            }

            Data data = 1;
            InternalData internal_data = 2;
            uint64 version = 3;
          }
        proto_message: MyMessage
    ...

Decoder with proto_import_paths:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: protobuf
      params:
        proto_file: 'example.proto'
        proto_message: MyMessage
        proto_import_paths:
          - path/to/proto_dir1
          - path/to/proto_dir2
    ...

The original event:

{
  "level": "error",
  "log": *proto binary message, for example 'MyMessage'*,
  "service": "test"
}

The resulting event:

{
  "level": "error",
  "service": "test",
  "data": {
	"string_data": "my_string",
	"int_data": 123
  },
  "internal_data": {
	"my_strings": ["str1","str2"],
	"is_valid": true
  },
  "version": 10
}
Syslog-RFC3164 decoder

The event root may contain any of the following fields:

  • priority string
  • facility string
  • severity string
  • timestamp string (Stamp format)
  • hostname string
  • app_name string
  • process_id string
  • message string

You can specify syslog_facility_format and syslog_severity_format in params for preferred facility and severity fields format.

Default decoder:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: syslog_rfc3164
    ...

The original event:

{
  "log": "<34>Oct  5 22:14:15 mymachine.example.com myproc[10]: 'myproc' failed on /dev/pts/8",
  "service": "test"
}

The resulting event:

{
  "service": "test",
  "priority": "34",
  "facility": "4",
  "severity": "2",
  "timestamp": "Oct  5 22:14:15",
  "hostname": "mymachine.example.com",
  "app_name": "myproc",
  "process_id": "10",
  "message": "'myproc' failed on /dev/pts/8"
}

Decoder with syslog_*_format params:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: syslog_rfc3164
      params:
        syslog_facility_format: 'string'
        syslog_severity_format: 'string'
    ...

The original event:

{
  "log": "<34>Oct 11 22:14:15 mymachine.example.com myproc: 'myproc' failed on /dev/pts/8",
  "service": "test"
}

The resulting event:

{
  "service": "test",
  "priority": "34",
  "facility": "AUTH",
  "severity": "CRIT",
  "timestamp": "Oct 11 22:14:15",
  "hostname": "mymachine.example.com",
  "app_name": "myproc",
  "message": "'myproc' failed on /dev/pts/8"
}
Syslog-RFC5424 decoder

The event root may contain any of the following fields:

  • priority string
  • facility string
  • severity string
  • proto_version string
  • timestamp string (RFC3339/RFC3339Nano format)
  • hostname string
  • app_name string
  • process_id string
  • message_id string
  • message string
  • SD_1 object
  • ...
  • SD_N object

You can specify syslog_facility_format and syslog_severity_format in params for preferred facility and severity fields format.

Default decoder:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: syslog_rfc5424
    ...

The original event:

{
  "log": "<165>1 2003-10-11T22:14:15.003Z mymachine.example.com myproc 10 ID47 [exampleSDID iut=\"3\" eventSource=\"Application\" eventID=\"1011\"] An application event log",
  "service": "test"
}

The resulting event:

{
  "service": "test",
  "priority": "165",
  "facility": "20",
  "severity": "5",
  "proto_version": "1",
  "timestamp": "2003-10-11T22:14:15.003Z",
  "hostname": "mymachine.example.com",
  "app_name": "myproc",
  "process_id": "10",
  "message_id": "ID47",
  "message": "An application event log",
  "exampleSDID": {
    "iut": "3",
    "eventSource": "Application",
    "eventID": "1011"
  }
}

Decoder with syslog_*_format params:

pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: syslog_rfc5424
      params:
        syslog_facility_format: 'string'
        syslog_severity_format: 'string'
    ...

The original event:

{
  "log": "<165>1 2003-10-11T22:14:15.003Z mymachine.example.com myproc - ID47 [exampleSDID iut=\"3\" eventSource=\"Application\" eventID=\"1011\"]",
  "service": "test"
}

The resulting event:

{
  "service": "test",
  "priority": "165",
  "facility": "LOCAL4",
  "severity": "NOTICE",
  "proto_version": "1",
  "timestamp": "2003-10-11T22:14:15.003Z",
  "hostname": "mymachine.example.com",
  "app_name": "myproc",
  "message_id": "ID47",
  "exampleSDID": {
    "iut": "3",
    "eventSource": "Application",
    "eventID": "1011"
  }
}
Keep origin
pipelines:
  example_pipeline:
    ...
    actions:
    - type: decode
      field: log
      decoder: json
      prefix: p_
      keep_origin: true
    ...

The original event:

{
  "level": "error",
  "log": "{\"message\":\"error occurred\",\"ts\":\"2023-10-30T13:35:33.638720813Z\",\"stream\":\"stderr\"}",
  "service": "test"
}

The resulting event:

{
  "level": "error",
  "log": "{\"message\":\"error occurred\",\"ts\":\"2023-10-30T13:35:33.638720813Z\",\"stream\":\"stderr\"}",
  "service": "test",
  "p_message": "error occurred",
  "p_ts": "2023-10-30T13:35:33.638720813Z",
  "p_stream": "stderr"
}

Config params

field cfg.FieldSelector required

The event field to decode. Must be a string.


decoder string default=json options=json|postgres|nginx_error|protobuf|syslog_rfc3164|syslog_rfc5424

Decoder type.


params map[string]any

Decoding params.

Json decoder params:

  • json_max_fields_size - map {path}: {limit} where {path} is path to field (cfg.FieldSelector) and {limit} is integer limit of the field length. If set, the fields will be cut to the specified limit.

    It works only with string values. If the field doesn't exist or isn't a string, it will be skipped.

Nginx-error decoder params:

  • nginx_with_custom_fields - if set, custom fields will be extracted.

Protobuf decoder params:

  • proto_file - protocol scheme, can be specified as both the path to the file and the contents of the file.
  • proto_message - message name in the specified proto_file.
  • proto_import_paths - optional list of paths within which the search will occur (including imports in proto_file). If present and not empty, then all file paths to find are assumed to be relative to one of these paths. Otherwise, all file paths to find are assumed to be relative to the current working directory.

If proto_file contains only system imports, then there is no need to add these files to one of the directories specified in proto_import_paths. Otherwise, all imports specified in the proto_file must be added to one of the directories specified in proto_import_paths respecting the file system tree.

List of system imports:

  • google/protobuf/any.proto
  • google/protobuf/api.proto
  • google/protobuf/compiler/plugin.proto
  • google/protobuf/descriptor.proto
  • google/protobuf/duration.proto
  • google/protobuf/empty.proto
  • google/protobuf/field_mask.proto
  • google/protobuf/source_context.proto
  • google/protobuf/struct.proto
  • google/protobuf/timestamp.proto
  • google/protobuf/type.proto
  • google/protobuf/wrappers.proto

Syslog-RFC3164 & Syslog-RFC5424 decoder params:

  • syslog_facility_format - facility format, must be one of number|string (number by default).
  • syslog_severity_format - severity format, must be one of number|string (number by default).

prefix string

Prefix to add to decoded keys.


keep_origin bool default=false

If set, the plugin will keep origin field after decoding.

If one of the decoded keys matches the original field, it will be overridden anyway.


log_decode_error_mode string default=off options=off|erronly|withnode

Defines how to handle logging of decode errors.

  • off – do not log decode errors
  • erronly – log only errors without any other data
  • withnode – log errors with field value


Generated using insane-doc

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {
	// > @3@4@5@6
	// >
	// > The event field to decode. Must be a string.
	Field  cfg.FieldSelector `json:"field" parse:"selector" required:"true"` // *
	Field_ []string

	// > @3@4@5@6
	// >
	// > Decoder type.
	Decoder  string `json:"decoder" default:"json" options:"json|postgres|nginx_error|protobuf|syslog_rfc3164|syslog_rfc5424"` // *
	Decoder_ decoderType

	// > @3@4@5@6
	// >
	// > Decoding params.
	// >
	// > **Json decoder params**:
	// > * `json_max_fields_size` - map `{path}: {limit}` where **{path}** is path to field (`cfg.FieldSelector`) and **{limit}** is integer limit of the field length.
	// > If set, the fields will be cut to the specified limit.
	// >	> It works only with string values. If the field doesn't exist or isn't a string, it will be skipped.
	// >
	// > **Nginx-error decoder params**:
	// > * `nginx_with_custom_fields` - if set, custom fields will be extracted.
	// >
	// > **Protobuf decoder params**:
	// > * `proto_file` - protocol scheme, can be specified as both the path to the file and the contents of the file.
	// > * `proto_message` - message name in the specified `proto_file`.
	// > * `proto_import_paths` - optional list of paths within which the search will occur (including imports in `proto_file`).
	// > If present and not empty, then all file paths to find are assumed to be relative to one of these paths. Otherwise, all file paths to find are assumed to be relative to the current working directory.
	// >> If `proto_file` contains only system imports, then there is no need to add these files to one of the directories specified in `proto_import_paths`.
	// >> Otherwise, all imports specified in the `proto_file` must be added to one of the directories specified in `proto_import_paths` respecting the file system tree.
	// >>
	// >> List of system imports:
	// >> * google/protobuf/any.proto
	// >> * google/protobuf/api.proto
	// >> * google/protobuf/compiler/plugin.proto
	// >> * google/protobuf/descriptor.proto
	// >> * google/protobuf/duration.proto
	// >> * google/protobuf/empty.proto
	// >> * google/protobuf/field_mask.proto
	// >> * google/protobuf/source_context.proto
	// >> * google/protobuf/struct.proto
	// >> * google/protobuf/timestamp.proto
	// >> * google/protobuf/type.proto
	// >> * google/protobuf/wrappers.proto
	// >
	// > **Syslog-RFC3164 & Syslog-RFC5424 decoder params**:
	// > * `syslog_facility_format` - facility format, must be one of `number|string` (`number` by default).
	// > * `syslog_severity_format` - severity format, must be one of `number|string` (`number` by default).
	Params map[string]any `json:"params"` // *

	// > @3@4@5@6
	// >
	// > Prefix to add to decoded keys.
	Prefix string `json:"prefix" default:""` // *

	// > @3@4@5@6
	// >
	// > If set, the plugin will keep origin `field` after decoding.
	// >
	// >> If one of the decoded keys matches the original `field`, it will be overridden anyway.
	KeepOrigin bool `json:"keep_origin" default:"false"` // *

	// > @3@4@5@6
	// >
	// > Defines how to handle logging of decode errors.
	// > @logDecodeErrorMode|comment-list
	LogDecodeErrorMode  string `json:"log_decode_error_mode" default:"off" options:"off|erronly|withnode"` // *
	LogDecodeErrorMode_ logDecodeErrorMode
}

! config-params ^ config-params

type Plugin 

type Plugin struct {
	// contains filtered or unexported fields
}

func (*Plugin) Do 

func (p *Plugin) Do(event *pipeline.Event) pipeline.ActionResult

func (*Plugin) Start 

func (p *Plugin) Start(config pipeline.AnyConfig, params *pipeline.ActionPluginParams)

func (*Plugin) Stop 

func (p *Plugin) Stop()

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.