slogproto

package module
v0.0.0-...-7607be9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 11, 2024 License: MPL-2.0 Imports: 18 Imported by: 0

README

slogproto

[!WARNING] This is an experimental module and is subject to change.

Go log/slog.Handler using Protocol Buffers. This can reduce the size of log messages when saving them to disk or sending them over the network, and can reduce the amount of time spent marshaling and unmarshaling log messages, at the cost of human readability.

To enable interopability with other tools, the slp CLI can read protobuf encoded slog.Records from STDIN (or a file) and output them as JSON to STDOUT. Logs can be filtered using CEL expressions.

Installation

For use with Go programs:

$ go get -u -v github.com/picatz/slogproto@latest

To use the slp CLI:

$ go install -v github.com/picatz/slogproto/cmd/slp@latest

Usage

package main

import (
	"log/slog"
	"os"

	"github.com/picatz/slogproto"
)

func main() {
	logger := slog.New(slogproto.NewHander(os.Stdout))

	logger.Info("example", slog.Int("something", 1))
}

Read from a program that produces slogproto formatted logs to STDOUT (like the example above):

$ go run main.go | slp
{"time":"2023-08-01T03:12:11.272826Z","level":"INFO","msg":"example","something":1}

Read from a file in slogproto format:

$ slp output.log
{"time":"..","level":"...","msg":"...", ... }
{"time":"..","level":"...","msg":"...", ... }
{"time":"..","level":"...","msg":"...", ... }
...

[!NOTE] Input to slp can be from STDIN or a file.

Filtering

The filter flag can be used to filter logs using a given CEL expression. The expression is evaluated against the slog.Record and must return a boolean value. For each log record that the expression evaluates as true will be output to STDOUT as JSON.

  • msg is the message in the log record.

  • level is the level in the log record.

  • time is the timestamp in the log record.

  • attrs is a map of all the attributes in the log record, not including the message, level, or time.

    attrs.something == 1

    has(attrs.something) && attrs.something == 1

    attrs.?something.orValue(0) == 1

    cel.bind(value, attrs.?something.else.orValue("default"), value != "example")

[!IMPORTANT] Invalid access to an attribute will cause the filter to fail at evaluation time. Invalid expressions (which do not evaluate to a boolean) will be checked before reading the log records, and will cause the program to exit with an error message.

$ slp --filter='has(attrs.something)' output.log
{"time":"2023-08-11T00:06:00.474782Z","level":"INFO","msg":"example","something":1}

$ slp --filter='msg == "this is a test"' test.log
{"time":"2023-08-11T00:06:00.474033Z","level":"INFO","msg":"this is a test","test":{"test2":"1","test3":1,"test1":1}}

File Format

The file format is a series of delimited Protocol Buffer messages. Each message is prefixed with a 32-bit unsigned integer representing the size of the message. The message itself is a protobuf encoded slog.Record.

╭────────────────────────────────────────────────────────────╮
│  Message Size  │  Protocol Buffer Message  │  ...  │  EOF  │
╰────────────────────────────────────────────────────────────╯

Comparisons to Other Formats

Using the following record written 1024 times:

{
	"time": $timestamp,
	"level": "INFO",
	"msg": "hello world",
	"i": $n
}
Format GZIP (KB) Snappy (KB) Zstandard (KB) Uncompressed (KB)
Protobuf 5.48 11.17 3.58 41.88
JSON 5.79 9.59 5.04 86.81
Text 2.93 7.66 1.31 69.92

Documentation

Overview

Package slogproto provides a protocol buffer definition for the slog format (golang.org/x/exp/slog).

It attempts to have minimial dependencies and minimize memory allocations.

Index

Examples

Constants

View Source 
const (
	LevelInfo  = Level_LEVEL_INFO
	LevelWarn  = Level_LEVEL_WARN
	LevelError = Level_LEVEL_ERROR
	LevelDebug = Level_LEVEL_DEBUG
)

Variables

View Source 
var (
	Level_name = map[int32]string{
		0: "LEVEL_UNSPECIFIED",
		1: "LEVEL_INFO",
		2: "LEVEL_WARN",
		3: "LEVEL_ERROR",
		4: "LEVEL_DEBUG",
	}
	Level_value = map[string]int32{
		"LEVEL_UNSPECIFIED": 0,
		"LEVEL_INFO":        1,
		"LEVEL_WARN":        2,
		"LEVEL_ERROR":       3,
		"LEVEL_DEBUG":       4,
	}
)

Enum value maps for Level.

View Source 
var File_slog_proto protoreflect.FileDescriptor

Functions

func CompileFilter 

func CompileFilter(expr string) (cel.Program, error)

CompileFilter compiles a filter expression into a program that can be evaluated against a slog record. The expression must evaluate to a boolean value: if it's true, the record should be included. It may reference the following variables:

  • msg: string
  • level: int
  • time: timestamp
  • attrs: map[string]any

The expression may also reference any of the functions provided by the CEL standard library, as well as the following functions provided by the CEL extension libraries:

  • strings
  • math
  • encoders
  • sets
  • lists
  • bindings

If the expression is invalid, an error is returned.

func EvalFilter 

func EvalFilter(prog cel.Program, r *slog.Record) (bool, error)

EvalFilter evaluates a filter program against a slog record. The record must be a map[string]any, and the program must have been compiled with CompileFilter. If the program is invalid, an error is returned.

The record must contain the following keys:

  • msg: string
  • level: int
  • time: timestamp
  • attrs: map[string]any

func Read 

func Read(ctx context.Context, r io.Reader, fn func(r *slog.Record) bool) error

Read reads protobuf encoded slog records from the reader and calls the provided function for each record. If the function returns false, the iteration is stopped.

If the context is canceled, the iteration is stopped and the error is returned. If the reader returns an error, the error is returned.

Types

type Handler 

type Handler struct {
	// contains filtered or unexported fields
}

Handler implements the slog.Handler interface and writes the log record to the writer as a protocol buffer encoded struct containing the log record, including the levem, message and attributes.

func NewHandler 

func NewHandler(w io.Writer, opts *slog.HandlerOptions) *Handler

NewHandler returns a new Handler that writes to the writer.

Example 

h := slogproto.NewHandler(os.Stdout)
Example 
package main

import (
	"bytes"
	"log/slog"

	"github.com/picatz/slogproto"
)

func main() {
	var logBuffer bytes.Buffer

	logger := slog.New(slogproto.NewHandler(&logBuffer, nil))

	logger.Info("this is a test",
		slog.Group("test",
			slog.Int("test1", 1),
			slog.String("test2", "1"),
			slog.Float64("test3", 1.0),
		),
	)

	logger.Info("example", slog.Int("something", 1))
}

Output:
Example (Gzip) 
package main

import (
	"bytes"
	"compress/gzip"
	"log/slog"

	"github.com/picatz/slogproto"
)

func main() {
	var logBuffer bytes.Buffer

	w := gzip.NewWriter(&logBuffer)

	logger := slog.New(slogproto.NewHandler(w, nil))

	logger.Info("hello world")

	err := w.Flush()
	if err != nil {
		panic(err)
	}

	err = w.Close()
	if err != nil {
		panic(err)
	}
}

Output:

func (*Handler) Enabled 

func (h *Handler) Enabled(ctx context.Context, level slog.Level) bool

Enabled returns true if the level is enabled for the handler.

func (*Handler) Handle 

func (h *Handler) Handle(ctx context.Context, r slog.Record) error

Handle writes the log record to the writer as a protocol buffer encoded struct containing the log record, including the level, message and attributes.

It will only be called when Enabled returns true. The Context argument is as for Enabled. It is present solely to provide Handlers access to the context's values. Canceling the context should not affect record processing. (Among other things, log messages may be necessary to debug a cancellation-related problem.)

Handle methods that produce output should observe the following rules:

  • If r.Time is the zero time, ignore the time.
  • If r.PC is zero, ignore it.
  • Attr's values should be resolved.
  • If an Attr's key and value are both the zero value, ignore the Attr. This can be tested with attr.Equal(Attr{}).
  • If a group's key is empty, inline the group's Attrs.
  • If a group has no Attrs (even if it has a non-empty key), ignore it.

func (*Handler) WithAttrs 

func (h *Handler) WithAttrs(attrs []slog.Attr) slog.Handler

WithAttrs returns a new Handler whose attributes consist of both the receiver's attributes and the arguments.

The Handler owns the slice: it may retain, modify or discard it.

func (*Handler) WithGroup 

func (h *Handler) WithGroup(name string) slog.Handler

WithGroup returns a new Handler with the given group appended to the receiver's existing groups.

The keys of all subsequent attributes, whether added by With or in a Record, should be qualified by the sequence of group names.

How this qualification happens is up to the Handler, so long as this Handler's attribute keys differ from those of another Handler with a different sequence of group names.

A Handler should treat WithGroup as starting a Group of Attrs that ends at the end of the log event. That is, 

logger.WithGroup("s").LogAttrs(level, msg, slog.Int("a", 1), slog.Int("b", 2))

should behave like 

logger.LogAttrs(level, msg, slog.Group("s", slog.Int("a", 1), slog.Int("b", 2)))

If the name is empty, WithGroup returns the receiver.

type Level 

type Level int32
const (
	Level_LEVEL_UNSPECIFIED Level = 0
	Level_LEVEL_INFO        Level = 1
	Level_LEVEL_WARN        Level = 2
	Level_LEVEL_ERROR       Level = 3
	Level_LEVEL_DEBUG       Level = 4
)

func (Level) Descriptor 

func (Level) Descriptor() protoreflect.EnumDescriptor

func (Level) Enum 

func (x Level) Enum() *Level

func (Level) EnumDescriptor deprecated 

func (Level) EnumDescriptor() ([]byte, []int)

Deprecated: Use Level.Descriptor instead.

func (Level) Number 

func (x Level) Number() protoreflect.EnumNumber

func (Level) String 

func (x Level) String() string

func (Level) Type 

func (Level) Type() protoreflect.EnumType

type Record 

type Record struct {
	Time    *timestamppb.Timestamp `protobuf:"bytes,1,opt,name=time,proto3" json:"time,omitempty"`
	Message string                 `protobuf:"bytes,2,opt,name=message,proto3" json:"message,omitempty"`
	Level   Level                  `protobuf:"varint,3,opt,name=level,proto3,enum=slog.Level" json:"level,omitempty"`
	Attrs   map[string]*Value      `` /* 151-byte string literal not displayed */
	// contains filtered or unexported fields
}

func (*Record) Descriptor deprecated 

func (*Record) Descriptor() ([]byte, []int)

Deprecated: Use Record.ProtoReflect.Descriptor instead.

func (*Record) GetAttrs 

func (x *Record) GetAttrs() map[string]*Value

func (*Record) GetLevel 

func (x *Record) GetLevel() Level

func (*Record) GetMessage 

func (x *Record) GetMessage() string

func (*Record) GetTime 

func (x *Record) GetTime() *timestamppb.Timestamp

func (*Record) ProtoMessage 

func (*Record) ProtoMessage()

func (*Record) ProtoReflect 

func (x *Record) ProtoReflect() protoreflect.Message

func (*Record) Reset 

func (x *Record) Reset()

func (*Record) String 

func (x *Record) String() string

type Value 

type Value struct {

	// Types that are assignable to Kind:
	//
	//	*Value_Bool
	//	*Value_Float
	//	*Value_Int
	//	*Value_String_
	//	*Value_Time
	//	*Value_Duration
	//	*Value_Uint
	//	*Value_Group_
	//	*Value_Any
	Kind isValue_Kind `protobuf_oneof:"kind"`
	// contains filtered or unexported fields
}

func (*Value) Descriptor deprecated 

func (*Value) Descriptor() ([]byte, []int)

Deprecated: Use Value.ProtoReflect.Descriptor instead.

func (*Value) GetAny 

func (x *Value) GetAny() *anypb.Any

func (*Value) GetBool 

func (x *Value) GetBool() bool

func (*Value) GetDuration 

func (x *Value) GetDuration() *durationpb.Duration

func (*Value) GetFloat 

func (x *Value) GetFloat() float64

func (*Value) GetGroup 

func (x *Value) GetGroup() *Value_Group

func (*Value) GetInt 

func (x *Value) GetInt() int64

func (*Value) GetKind 

func (m *Value) GetKind() isValue_Kind

func (*Value) GetString_ 

func (x *Value) GetString_() string

func (*Value) GetTime 

func (x *Value) GetTime() *timestamppb.Timestamp

func (*Value) GetUint 

func (x *Value) GetUint() uint64

func (*Value) ProtoMessage 

func (*Value) ProtoMessage()

func (*Value) ProtoReflect 

func (x *Value) ProtoReflect() protoreflect.Message

func (*Value) Reset 

func (x *Value) Reset()

func (*Value) String 

func (x *Value) String() string

type Value_Any 

type Value_Any struct {
	Any *anypb.Any `protobuf:"bytes,9,opt,name=any,proto3,oneof"`
}

type Value_Bool 

type Value_Bool struct {
	Bool bool `protobuf:"varint,1,opt,name=bool,proto3,oneof"`
}

type Value_Duration 

type Value_Duration struct {
	Duration *durationpb.Duration `protobuf:"bytes,6,opt,name=duration,proto3,oneof"`
}

type Value_Float 

type Value_Float struct {
	Float float64 `protobuf:"fixed64,2,opt,name=float,proto3,oneof"`
}

type Value_Group 

type Value_Group struct {
	Attrs map[string]*Value `` /* 151-byte string literal not displayed */
	// contains filtered or unexported fields
}

func (*Value_Group) Descriptor deprecated 

func (*Value_Group) Descriptor() ([]byte, []int)

Deprecated: Use Value_Group.ProtoReflect.Descriptor instead.

func (*Value_Group) GetAttrs 

func (x *Value_Group) GetAttrs() map[string]*Value

func (*Value_Group) ProtoMessage 

func (*Value_Group) ProtoMessage()

func (*Value_Group) ProtoReflect 

func (x *Value_Group) ProtoReflect() protoreflect.Message

func (*Value_Group) Reset 

func (x *Value_Group) Reset()

func (*Value_Group) String 

func (x *Value_Group) String() string

type Value_Group_ 

type Value_Group_ struct {
	Group *Value_Group `protobuf:"bytes,8,opt,name=group,proto3,oneof"`
}

type Value_Int 

type Value_Int struct {
	Int int64 `protobuf:"varint,3,opt,name=int,proto3,oneof"`
}

type Value_String_ 

type Value_String_ struct {
	String_ string `protobuf:"bytes,4,opt,name=string,proto3,oneof"`
}

type Value_Time 

type Value_Time struct {
	Time *timestamppb.Timestamp `protobuf:"bytes,5,opt,name=time,proto3,oneof"`
}

type Value_Uint 

type Value_Uint struct {
	Uint uint64 `protobuf:"varint,7,opt,name=uint,proto3,oneof"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
slp

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.