logtree

package
v0.0.0-...-5fb8a3f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 30, 2024 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package logtree implements a tree-shaped logger for debug events. It provides log publishers (ie. Go code) with a glog-like API and io.Writer API, with loggers placed in a hierarchical structure defined by a dot-delimited path (called a DN, short for Distinguished Name). 

tree.MustLeveledFor("foo.bar.baz").Warningf("Houston, we have a problem: %v", err)
fmt.Fprintf(tree.MustRawFor("foo.bar.baz"), "some\nunstructured\ndata\n")

Logs in this context are unstructured, operational and developer-centric human readable text messages presented as lines of text to consumers, with some attached metadata. Logtree does not deal with 'structured' logs as some parts of the industry do, and instead defers any machine-readable logs to either be handled by metrics systems like Prometheus or event sourcing systems like Kafka.

Tree Structure

As an example, consider an application that produces logs with the following DNs: 

listener.http
listener.grpc
svc
svc.cache
svc.cache.gc

This would correspond to a tree as follows: 

                      .------.
                     |   ""   |
                     | (root) |
                      '------'
       .----------------'   '------.
.--------------.           .---------------.
|     svc      |           |    listener   |
'--------------'           '---------------'
       |                   .----'      '----.
.--------------.  .---------------.  .---------------.
|  svc.cache   |  | listener.http |  | listener.grpc |
'--------------'  '---------------'  '---------------'
       |
.--------------.
| svc.cache.gc |
'--------------'

In this setup, every DN acts as a separate logging target, each with its own retention policy and quota. Logging to a DN under foo.bar does NOT automatically log to foo - all tree mechanisms are applied on log access by consumers. Loggers are automatically created on first use, and importantly, can be created at any time, and will automatically be created if a sub-DN is created that requires a parent DN to exist first. Note, for instance, that a `listener` logging node was created even though the example application only logged to `listener.http` and `listener.grpc`.

An implicit root node is always present in the tree, accessed by DN "" (an empty string). All other logger nodes are children (or transitive children) of the root node.

Log consumers (application code that reads the log and passes them on to operators, or ships them off for aggregation in other systems) to select subtrees of logs for readout. In the example tree, a consumer could select to either read all logs of the entire tree, just a single DN (like svc), or a subtree (like everything under listener, ie. messages emitted to listener.http and listener.grpc).

Leveled Log Producer API

As part of the glog-like logging API available to producers, the following metadata is attached to emitted logs in addition to the DN of the logger to which the log entry was emitted:

  • timestamp at which the entry was emitted
  • a severity level (one of FATAL, ERROR, WARN or INFO)
  • a source of the message (file name and line number)

In addition, the logger mechanism supports a variable verbosity level (so-called 'V-logging') that can be set at every node of the tree. For more information about the producer-facing logging API, see the documentation of the LeveledLogger interface, which is the main interface exposed to log producers.

If the submitted message contains newlines, it will be split accordingly into a single log entry that contains multiple string lines. This allows for log producers to submit long, multi-line messages that are guaranteed to be non-interleaved with other entries, and allows for access API consumers to maintain semantic linking between multiple lines being emitted as a single atomic entry.

Raw Log Producer API

In addition to leveled, glog-like logging, LogTree supports 'raw logging'. This is implemented as an io.Writer that will split incoming bytes into newline-delimited lines, and log them into that logtree's DN. This mechanism is primarily intended to support storage of unstructured log data from external processes - for example binaries running with redirected stdout/stderr.

Log Access API

The Log Access API is mostly exposed via a single function on the LogTree struct: Read. It allows access to log entries that have been already buffered inside LogTree and to subscribe to receive future entries over a channel. As outlined earlier, any access can specify whether it is just interested in a single logger (addressed by DN), or a subtree of loggers.

Due to the current implementation of the logtree, subtree accesses of backlogged data is significantly slower than accessing data of just one DN, or the whole tree (as every subtree backlog access performs a scan on all logged data). Thus, log consumers should be aware that it is much better to stream and buffer logs specific to some long-standing logging request on their own, rather than repeatedly perform reads of a subtree backlog.

The data returned from the log access API is a LogEntry, which itself can contain either a raw logging entry, or a leveled logging entry. Helper functions are available on LogEntry that allow canonical string representations to be returned, for easy use in consuming tools/interfaces. Alternatively, the consumer can itself access the internal raw/leveled entries and print them according to their own preferred format.

Index

Constants

View Source 
const BacklogAllAvailable int = -1

BacklogAllAvailable makes WithBacklog return all backlogged log data that logtree possesses.

Variables

View Source 
var (
	ErrInvalidDN = errors.New("invalid DN")
)
View Source 
var (
	ErrRawAndLeveled = errors.New("cannot return logs that are simultaneously OnlyRaw and OnlyLeveled")
)
View Source 
var MetropolisShortenDict = ShortenDictionary{
	"controlplane":           "cplane",
	"map-cluster-membership": "map-membership",
	"cluster-membership":     "cluster",
	"controller-manager":     "controllers",
	"networking":             "net",
	"network":                "net",
	"interfaces":             "ifaces",
	"kubernetes":             "k8s",
}

Functions

func GRPCify 

func GRPCify(logger logging.Leveled) grpclog.LoggerV2

GRPCify turns a LeveledLogger into a go-grpc compatible logger.

func KLogParser 

func KLogParser(logger logging.Leveled) io.WriteCloser

KLogParser returns an io.WriteCloser to which raw logging from a klog emitter can be piped. It will attempt to parse all lines from this log as glog/klog-style entries, and pass them over to a LeveledLogger as if they were emitted locally.

This allows for piping in external processes that emit klog logging into a logtree, leading to niceties like transparently exposing the log severity or source file/line.

One caveat, however, is that V-leveled logs will not be translated appropriately - anything that the klog-emitter pumps out as Info will be directly ingested as Info logging. There is no way to work around this.

Another important limitation is that any written line is interpreted as having happened recently (ie. within one hour of the time of execution of this function). This is important as klog/glog-formatted loglines don't have a year attached, so we have to infer it based on the current timestamp (note: parsed lines do not necessarily have their year aleays equal to the current year, as the code handles the edge case of parsing a line from the end of a previous year at the beginning of the next).

func KmsgPipe 

func KmsgPipe(ctx context.Context, lt logging.Leveled) error

KmsgPipe pipes logs from the kernel kmsg interface at /dev/kmsg into the given logger.

func LogExternalLeveled 

func LogExternalLeveled(l logging.Leveled, e *ExternalLeveledPayload) error

LogExternalLeveled injects a ExternalLeveledPayload into a given LeveledLogger. This should only be used by systems which translate external data sources into leveled logging - see ExternelLeveledPayload for more information.

func PipeAllToTest 

func PipeAllToTest(t testing.TB, lt *LogTree)

PipeAllToTest starts a goroutine that will forward all logtree entries t.Logf(), in the canonical logtree payload representation.

It's designed to be used in tests, and will automatically stop when the test/benchmark it's running in exits.

func SeverityFromProto 

func SeverityFromProto(s lpb.LeveledLogSeverity) (logging.Severity, error)

func SeverityToProto 

func SeverityToProto(s logging.Severity) lpb.LeveledLogSeverity

func Zapify 

func Zapify(logger logging.Leveled, minimumLevel zapcore.Level) *zap.Logger

Zapify turns a LeveledLogger into a zap.Logger which pipes its output into the LeveledLogger. The message, severity and caller are carried over. Extra fields are appended as JSON to the end of the log line.

Types

type DN 

type DN string

DN is the Distinguished Name, a dot-delimited path used to address loggers within a LogTree. For example, "foo.bar" designates the 'bar' logger node under the 'foo' logger node under the root node of the logger. An empty string is the root node of the tree.

func (DN) Path 

func (d DN) Path() ([]string, error)

Path return the parts of a DN, ie. all the elements of the dot-delimited DN path. For the root node, an empty list will be returned. An error will be returned if the DN is invalid (contains empty parts, eg. `foo..bar`, `.foo` or `foo.`.

func (DN) Shorten 

func (d DN) Shorten(dict ShortenDictionary, maxLen int) string

Shorten returns a shortened version of this DN for constrained logging environments like tty0 logging.

If ShortenDictionary is given, it will be used to replace DN parts with shorter equivalents. For example, with the dictionary:

{ "foobar": "foo", "manager": "mgr" }

The DN some.foobar.logger will be turned into some.foo.logger before further being processed by the shortening mechanism.

The shortening rules applied are Metropolis-specific.

type ExternalLeveledPayload 

type ExternalLeveledPayload struct {
	// Log line. If any newlines are found, they will split the message into
	// multiple messages within LeveledPayload. Empty messages are accepted
	// verbatim.
	Message string
	// Timestamp when this payload was emitted according to its source. If not
	// given, will default to the time of conversion to LeveledPayload.
	Timestamp time.Time
	// Log severity. If invalid or unset will default to INFO.
	Severity logging.Severity
	// File name of originating code. Defaults to "unknown" if not set.
	File string
	// Line in File. Zero indicates the line is not known.
	Line int
}

ExternalLeveledPayload is a LeveledPayload received from an external source, eg. from parsing the logging output of third-party programs. It can be converted into a LeveledPayload and inserted into a leveled logger, but will be sanitized before that, ensuring that potentially buggy emitters/converters do not end up polluting the leveled logger data.

This type should be used only when inserting data from external systems, not by code that just wishes to log things. In the future, data inserted this way might be explicitly marked as tainted so operators can understand that parts of this data might not give the same guarantees as the log entries emitted by the native LeveledLogger API.

type LeveledPayload 

type LeveledPayload struct {
	// contains filtered or unexported fields
}

LeveledPayload is a log entry for leveled logs (as per leveled.go). It contains the input to these calls (severity and message split into newline-delimited messages) and additional metadata that would be usually seen in a text representation of a leveled log entry.

func LeveledPayloadFromProto 

func LeveledPayloadFromProto(p *lpb.LogEntry_Leveled) (*LeveledPayload, error)

LeveledPayloadFromProto parses a protobuf message into the internal format.

func (*LeveledPayload) Location 

func (p *LeveledPayload) Location() string

Location returns a string in the form of file_name:line_number that shows the origin of the log entry in the program source.

func (*LeveledPayload) Messages 

func (p *LeveledPayload) Messages() []string

Messages returns the inner message lines of this entry, ie. what was passed to the actual logging method, but split by newlines.

func (*LeveledPayload) MessagesJoined 

func (p *LeveledPayload) MessagesJoined() string

func (*LeveledPayload) Proto 

func (p *LeveledPayload) Proto() *lpb.LogEntry_Leveled

Proto converts a LeveledPayload to protobuf format.

func (*LeveledPayload) Severity 

func (p *LeveledPayload) Severity() logging.Severity

Severity returns the Severity with which this entry was logged.

func (*LeveledPayload) String 

func (p *LeveledPayload) String() string

String returns a canonical representation of this payload as a single string prefixed with metadata. If the original message was logged with newlines, this representation will also contain newlines, with each original message part prefixed by the metadata. For an alternative call that will instead return a canonical prefix and a list of lines in the message, see Strings().

func (*LeveledPayload) Strings 

func (p *LeveledPayload) Strings() (prefix string, lines []string)

Strings returns the canonical representation of this payload split into a prefix and all lines that were contained in the original message. This is meant to be displayed to the user by showing the prefix before each line, concatenated together - possibly in a table form with the prefixes all unified with a rowspan- like mechanism.

For example, this function can return: 

prefix = "I1102 17:20:06.921395 foo.go:42] "
lines = []string{"current tags:", " - one", " - two"}

With this data, the result should be presented to users this way in text form: I1102 17:20:06.921395 foo.go:42] current tags: I1102 17:20:06.921395 foo.go:42] - one I1102 17:20:06.921395 foo.go:42] - two

Or, in a table layout: .-----------------------------------------------------------. | I1102 17:20:06.921395 0 foo.go:42] : current tags: | | :------------------| | : - one | | :------------------| | : - two | '-----------------------------------------------------------'

func (*LeveledPayload) Timestamp 

func (p *LeveledPayload) Timestamp() time.Time

Timestamp returns the time at which this entry was logged.

type LogEntry 

type LogEntry struct {
	// If non-nil, this is a leveled logging entry.
	Leveled *LeveledPayload
	// If non-nil, this is a raw logging entry line.
	Raw *logbuffer.Line
	// DN from which this entry was logged.
	DN DN
}

LogEntry contains a log entry, combining both leveled and raw logging into a single stream of events. A LogEntry will contain exactly one of either LeveledPayload or RawPayload.

func LogEntryFromProto 

func LogEntryFromProto(l *lpb.LogEntry) (*LogEntry, error)

LogEntryFromProto parses a proto LogEntry back into internal structure. This can be used in log proto API consumers to easily print received log entries.

func (*LogEntry) ConciseString 

func (l *LogEntry) ConciseString(dict ShortenDictionary, maxWidth int) string

ConciseString returns a concise representation of this log entry for constrained environments, like TTY consoles.

The output format is as follows: 

  shortened dn I Hello there
some component W Something went wrong
  shortened dn I Goodbye there
external stuff R I am en external process using raw logging.

The above output is the result of calling ConciseString on three different LogEntries.

If maxWidth is greater than zero, word wrapping will be applied. For example, with maxWidth set to 40: 

     shortened I Hello there
some component W Something went wrong and here are the very long details that
               | describe this particular issue: according to all known laws of
               | aviation, there is no way a bee should be able to fly.
  shortened dn I Goodbye there
external stuff R I am en external process using raw logging.

The above output is also the result of calling ConciseString on three different LogEntries.

Multi-line log entries will emit 'continuation' lines (with '|') in the same way as word wrapping does. That means that even with word wrapping disabled, the result of this function might be multiline.

The width of the first column (the 'shortened DN' column) is automatically selected based on maxWidth. If maxWidth is less than 60, the column will be omitted. For example, with maxWidth set to 20: 

I Hello there
W Something went wrong and here are the very long details that
| describe this particular issue: according to all known laws of
| aviation, there is no way a bee should be able to fly.
I Goodbye there
R I am en external process using raw logging.

The given `dict` implements simple replacement rules for shortening the DN parts of a log entry's DN. Some rules are hardcoded for Metropolis' DN tree. If no extra shortening rules should be applied, dict can be set to ni// The given `dict` implements simple replacement rules for shortening the DN parts of a log entry's DN. Some rules are hardcoded for Metropolis' DN tree. If no extra shortening rules should be applied, dict can be set to nil.

func (*LogEntry) Proto 

func (l *LogEntry) Proto() *lpb.LogEntry

Proto converts this LogEntry to proto. Returned value may be nil if given LogEntry is invalid, eg. contains neither a Raw nor Leveled entry.

func (*LogEntry) String 

func (l *LogEntry) String() string

String returns a canonical representation of this payload as a single string prefixed with metadata. If the entry is a leveled log entry that originally was logged with newlines this representation will also contain newlines, with each original message part prefixed by the metadata. For an alternative call that will instead return a canonical prefix and a list of lines in the message, see Strings().

func (*LogEntry) Strings 

func (l *LogEntry) Strings() (prefix string, lines []string)

type LogReadOption 

type LogReadOption struct {
	// contains filtered or unexported fields
}

LogReadOption describes options for the LogTree.Read call.

func LeveledWithMinimumSeverity 

func LeveledWithMinimumSeverity(s logging.Severity) LogReadOption

LeveledWithMinimumSeverity makes Read return only log entries that are at least at a given Severity. If only leveled entries are needed, OnlyLeveled must be used. This is a no-op when OnlyRaw is used.

func OnlyLeveled 

func OnlyLeveled() LogReadOption

func OnlyRaw 

func OnlyRaw() LogReadOption

func WithBacklog 

func WithBacklog(count int) LogReadOption

WithBacklog makes Read return already recorded log entries, up to count elements.

func WithChildren 

func WithChildren() LogReadOption

WithChildren makes Read return/stream data for both a given DN and all its children.

func WithStream 

func WithStream() LogReadOption

WithStream makes Read return a stream of data. This works alongside WithBacklog to create a read-and-stream construct.

type LogReader 

type LogReader struct {
	// Backlog are the log entries already logged by LogTree. This will only be set if
	// WithBacklog has been passed to Read.
	Backlog []*LogEntry
	// Stream is a channel of new entries as received live by LogTree. This will only
	// be set if WithStream has been passed to Read. In this case, entries from this
	// channel must be read as fast as possible by the consumer in order to prevent
	// missing entries.
	Stream <-chan *LogEntry
	// contains filtered or unexported fields
}

LogReader permits reading an already existing backlog of log entries and to stream further ones.

func (*LogReader) Close 

func (l *LogReader) Close()

Close closes the LogReader's Stream. This must be called once the Reader does not wish to receive streaming messages anymore.

func (*LogReader) Missed 

func (l *LogReader) Missed() uint64

Missed returns the amount of entries that were missed from Stream (as the channel was not drained fast enough).

type LogTree 

type LogTree struct {
	// contains filtered or unexported fields
}

LogTree is a tree-shaped logging system. For more information, see the package- level documentation.

func New 

func New() *LogTree

func (*LogTree) LeveledFor 

func (l *LogTree) LeveledFor(dn DN) (logging.Leveled, error)

LeveledFor returns a LeveledLogger publishing interface for a given DN. An error may be returned if the DN is malformed.

func (*LogTree) MustLeveledFor 

func (l *LogTree) MustLeveledFor(dn DN) logging.Leveled

MustLeveledFor returns a LeveledLogger publishing interface for a given DN, or panics if the given DN is invalid.

func (*LogTree) MustRawFor 

func (l *LogTree) MustRawFor(dn DN) io.Writer

func (*LogTree) RawFor 

func (l *LogTree) RawFor(dn DN) (io.Writer, error)

func (*LogTree) Read 

func (l *LogTree) Read(dn DN, opts ...LogReadOption) (*LogReader, error)

Read and/or stream entries from a LogTree. The returned LogReader is influenced by the LogReadOptions passed, which influence whether the Read will return existing entries, a stream, or both. In addition the options also dictate whether only entries for that particular DN are returned, or for all sub-DNs as well.

func (*LogTree) SetVerbosity 

func (l *LogTree) SetVerbosity(dn DN, level logging.VerbosityLevel) error

SetVerbosity sets the verbosity for a given DN (non-recursively, ie. for that DN only, not its children).

type ShortenDictionary 

type ShortenDictionary map[string]string

Source Files

View all Source files

Directories

Path Synopsis
Package unraw implements a facility to convert raw logs from external sources into leveled logs.
 Package unraw implements a facility to convert raw logs from external sources into leveled logs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.