encoding

README

logplex/encoding

What's this?

A set of libraries we use to parse messages, and to also publish these same syslog RFC5424 messages.

How to use?

We have 2 scanners available. If you're trying to build a logplex compatible ingress, you can use the regular scanner.

Scanner

func handler(w http.ResponseWriter, r *http.Request) {
	s := NewScanner(r.Body)

	for s.Scan() {
		log.Printf("%+v", scanner.Message())
	}

	if s.Err() != nil {
		log.Printf("err: %v", s.Err())
	}
}

DrainScanner

If the intent is to write an application which acts as a heroku drain, then using the DrainScanner is preferrable -- primarily because it doesn't require structured data.

func handler(w http.ResponseWriter, r *http.Request) {
	s := NewDrainScanner(r.Body)

	for s.Scan() {
		log.Printf("%+v", scanner.Message())
	}

	if s.Err() != nil {
		log.Printf("err: %v", s.Err())
	}
}

Documentation

Index

• Constants
• Variables
• func Encode(msg Message) ([]byte, error)
• func SyslogSplitFunc(data []byte, atEOF bool) (advance int, token []byte, err error)
• func TruncatingSyslogSplitFunc(maxLength int) bufio.SplitFunc
• type Encoder
  • func NewPlain(w io.Writer) Encoder
  • func NewSSE(w io.Writer) Encoder
• type Message
  • func Decode(raw []byte, hasStructuredData bool) (Message, error)
  • func (m Message) Size() (int, error)
  • func (m Message) WriteTo(w io.Writer) (int64, error)
• type Scanner
  • func NewDrainScanner(r io.Reader, opts ...ScannerOption) Scanner
  • func NewScanner(r io.Reader, opts ...ScannerOption) Scanner
• type ScannerOption
  • func RFCCompliant(compliant bool) ScannerOption
  • func WithBuffer(optimalFrameLength, maxFrameLength int) ScannerOption
  • func WithSplit(splitFunc bufio.SplitFunc) ScannerOption

Constants

const (
	// MaxFrameLength is the maximum message size to parse
	MaxFrameLength = 10240

	// OptimalFrameLength is the initial buffer size for scanning
	OptimalFrameLength = 1024
)

const FlexibleSyslogTimeFormat = "2006-01-02T15:04:05.999999Z07:00"

FlexibleSyslogTimeFormat accepts both 'Z' and TZ notation for event time.

const HumanTimeFormat = "2006-01-02T15:04:05.000000-07:00"

HumanTimeFormat defines the human friendly time format used in CLI/UI.

const L15Error = "L15: Error displaying log lines. Please try again."

L15Error is the message returned with an L15 error

const SyslogTimeFormat = "2006-01-02T15:04:05.999999-07:00"

SyslogTimeFormat defines the exact time format used in our logs.

Variables

var (
	// ErrBadFrame is returned when the scanner cannot parse syslog message boundaries
	ErrBadFrame = errors.New("bad frame")

	// ErrInvalidStructuredData is returned when structure data has any value other than '-' (blank)
	ErrInvalidStructuredData = errors.New("invalid structured data")

	// ErrInvalidPriVal is returned when pri-val is not properly formatted
	ErrInvalidPriVal = errors.New("invalid pri-val")
)

var ErrInvalidMessage = errors.New("invalid message")

ErrInvalidMessage returned when trying to encode an invalid syslog message

Functions

func Encode

func Encode(msg Message) ([]byte, error)

Encode serializes a syslog message into their wire format ( octet-framed syslog ) Disabling RFC 5424 compliance is the default and needed due to https://github.com/heroku/logplex/issues/204

func SyslogSplitFunc

func SyslogSplitFunc(data []byte, atEOF bool) (advance int, token []byte, err error)

SyslogSplitFunc splits the data based on the defined length prefix. format: 64 <190>1 2019-07-20T17:50:10.879238Z shuttle token shuttle - - 99\n65 <190>1 2019-07-20T17:50:10.879238Z shuttle token shuttle - - 100\n ^ frame size ^ boundary

func TruncatingSyslogSplitFunc

func TruncatingSyslogSplitFunc(maxLength int) bufio.SplitFunc

TruncatingSyslogSplitFunc enforces a maximum line length after parsing.

Types

type Encoder

type Encoder interface {
	Encode(msg Message) error
	KeepAlive() error
}

Encoder abstracts away how messages are written out

func NewPlain

func NewPlain(w io.Writer) Encoder

NewPlain creates a plain encoder. It dumps the log message directly without massaging it

func NewSSE

func NewSSE(w io.Writer) Encoder

NewSSE instantiates a new SSE encoder

type Message

type Message struct {
	Timestamp    time.Time
	Hostname     string
	Application  string
	Process      string
	ID           string
	Message      string
	Version      uint16
	Priority     uint8
	RFCCompliant bool
}

Message is a syslog message

func Decode

func Decode(raw []byte, hasStructuredData bool) (Message, error)

Decode converts a rfc5424 message to our model

func (Message) Size

func (m Message) Size() (int, error)

Size returns the message size in bytes, including the octet framing header

func (Message) WriteTo

func (m Message) WriteTo(w io.Writer) (int64, error)

WriteTo writes the message to a stream

type Scanner

type Scanner interface {
	Scan() bool
	Err() error
	Message() Message
}

Scanner is the general purpose primitive for parsing message bodies coming from log-shuttle, logfwd, logplex and all sorts of logging components.

func NewDrainScanner

func NewDrainScanner(r io.Reader, opts ...ScannerOption) Scanner

NewDrainScanner returns a scanner for use with drain endpoints. The primary difference is that it's loose and doesn't check for structured data.

func NewScanner

func NewScanner(r io.Reader, opts ...ScannerOption) Scanner

NewScanner is a syslog octet frame stream parser

type ScannerOption

type ScannerOption func(*syslogScanner)

func RFCCompliant

func RFCCompliant(compliant bool) ScannerOption

func WithBuffer

func WithBuffer(optimalFrameLength, maxFrameLength int) ScannerOption

func WithSplit

func WithSplit(splitFunc bufio.SplitFunc) ScannerOption