syslog2nats

README

syslog2nats

Receives syslog messages and publishs them to NATS

syslog2nats is based on

• syslogsidecar
• sputnik

syslog2nats consists of:

• syslog server - common part for all syslogsidecar based processes
• NATS specific plugins

Syslog server

Supported RFCs:

• RFC3164
• RFC5424

RFC3164 message consists of following symbolic parts:

• priority
• facility
• severity
• timestamp
• hostname
• tag
• content

RFC5424

RFC5424 message consists of following symbolic parts:

• priority
• facility
• severity
• timestamp
• hostname
• version
• app_name
• proc_id
• msg_id
• structured_data
• message

Non-RFC parts

syslogsidecar adds rfc of produced message:

• Part name: "rfc"
• Values: "RFC3164"|"RFC5424"

Badly formatted messages

syslogsidecar creates only one part for badly formatted message - former syslog message:

• Part name: "data"

Severities

Valid severity levels and names are:

• 0 emerg
• 1 alert
• 2 crit
• 3 err
• 4 warning
• 5 notice
• 6 info
• 7 debug

syslogsidecar filters messages by level according to value in configuration, e.g. for:

{
  "SEVERITYLEVEL": 4,
  ...........
}

all messages with severity above 4 will be discarded.

Configuration

Configuration of syslog server part of syslogsidecar is saved in the file syslogreceiver.json:

{
    "SEVERITYLEVEL": 4,
    "ADDRTCP": "127.0.0.1:5141",
    "ADDRUDP": "127.0.0.1:5141",
    "UDSPATH": "",
    "ADDRTCPTLS": "127.0.0.1:5143",
    "CLIENT_CERT_PATH": "",
    "CLIENT_KEY_PATH ": "",
    "ROOT_CA_PATH": ""
}

Links

• More complete description of syslogsidecar
• syslog for Memphis is part of memphis-protocol-adapter project

Plugins

NATS plugins to syslogsidecar:

• connector
• producer
• consumer (used for the tests)

Connector

Configuration file: connector.json

{
    "CONN_URL": "nats://127.0.0.1:4222"
}

The rest of connection options are default.

Connector creates sharable nats.Conn for:

• periodic validation of connectivity with memphis
• using by producer (production) and consumer (e2e test)

More about connector and underlying software - sputnik

Producer

Configuration file: syslogproducer.json

{
    "STREAM": "syslog"
}

Producer uses received from connector nats.Conn. It created JETSTREAM with name from configuration, the rest of stream options are default.

syslog messages are produced to jetstream as Header with empty payload:

    .................................
    msg := &nats.Msg{
		Subject: name,
		Header:  make(nats.Header),
	}

	putToheader := func(name string, value string) error {
		msg.Header.Add(name, value)
		return nil
	}

	syslogsidecar.Unpack(inmsg, putToheader)
    .................................

Build and run under vscode

go clean -cache -testcache
go build ./cmd/syslog2nats/
./syslog2nats -cf ./cmd/syslog2nats/conf/

e2e test

Simultaneuosly:

• send 1000000 syslog messages
  • TCP/IP
  • RFC5424
• receive
• produce
• consume
• compare

Build and run under vscode:

go clean -cache -testcache
go build ./cmd/syslog-e2e/
./syslog-e2e -cf ./cmd/syslog-e2e/conf/

nats server runs as embedded within syslog-e2e process.

Documentation

Index

• Constants
• func ConvertConsumeMsg(inmsg jetstream.Msg) sputnik.Msg
• func ConvertProduceMsg(name string, inmsg sputnik.Msg) *nats.Msg
• func NewConnector() sputnik.ServerConnector
• func RunBasicJetStreamServer(port int) *server.Server
• func RunServerWithOptions(opts server.Options) *server.Server
• func ShutdownJSServerAndRemoveStorage(s *server.Server)
• type MsgPrdConfig
• type NatsConnection

Constants

const (
	NATSPORT = 4222
	CONFPATH = "./conf"
)

const MsgConsumerConfigName = MsgProducerConfigName

const MsgProducerConfigName = syslogsidecar.ProducerName

Functions

func ConvertConsumeMsg

func ConvertConsumeMsg(inmsg jetstream.Msg) sputnik.Msg

func ConvertProduceMsg

func ConvertProduceMsg(name string, inmsg sputnik.Msg) *nats.Msg

func NewConnector

func NewConnector() sputnik.ServerConnector

func RunBasicJetStreamServer

func RunBasicJetStreamServer(port int) *server.Server

func RunServerWithOptions

func RunServerWithOptions(opts server.Options) *server.Server

func ShutdownJSServerAndRemoveStorage

func ShutdownJSServerAndRemoveStorage(s *server.Server)

Types

type MsgPrdConfig

type MsgPrdConfig struct {
	STREAM string
}

type NatsConnection

type NatsConnection struct {
	Shared   bool
	NatsConn *nats.Conn
}