Documentation
¶
Overview ¶
Package nanlogger collects `graph.Node` objects to monitor for `NaN` ("not-a-number") or `Inf` (infinity) values.
It does that by implementing `graph.LoggerFn` and hooking to the `graph.Exec` that executes the graph -- it can also `AttachToTrainer`, so it automatically attaches to every graph the trainer creates.
If at the end of a graph.Exec call, if a `NaN` value is found on the traced computation nodes, the first node where it appears (often `NaN` values spread through the graph) is reported back.
The report includes a stack trace and an optional user set scoped context.
Example: create a `NanLogger` and attaches it to the trainer, to it gets attached to every graph created (if more than one is created by the trainer).
func train() {
…
nanLogger := nanlogger.New()
trainer := train.NewTrainer(…)
nanLogger.AttachToTrainer(trainer)
…
}
func ModelGraph(ctx *context.Context, spec any, inputs []*Node) []*Node {
…
for ii := 0; ii < numBlocks; ii++ {
x = ResidualBlock(ctx.In(name), x, lastNumChannels)
nanLogger.Trace(x, fmt.Sprintf("Residual-%d", ii+1))
}
…
}
Index ¶
Constants ¶
const UniqueMessageId = "#nanlogger"
Variables ¶
This section is empty.
Functions ¶
func DefaultHandler ¶
DefaultHandler when a `NaN` or `Inf` is observed: it prints all out all the information and exits.
Types ¶
type ExecWithLogger ¶
type ExecWithLogger interface {
SetNodeLogger(loggerFn graph.LoggerFn)
GetNodeLogger() graph.LoggerFn
}
ExecWithLogger represents any of the executors in GoMLX (or future): `graph.Exec` and `context.Exec`. What is required is that it supports setting the logger and reading the current logger.
type NanLogger ¶
type NanLogger struct {
// contains filtered or unexported fields
}
NanLogger uses the logger infrastructure to monitor for NaN (and Inf) values in your graph. You manually select the nodes you want to monitor, and it saves the stack where it was called along with user provided scope information. If during the execution any NaN appears, it panics with the stack trace where the monitor was set, along with the scope. Alternatively, instead of panicking, one can set a handler to be called if/when a NaN is observed.
See example in package documentation.
func New ¶
func New() *NanLogger
New creates a NanLogger that can be used to debug where NaN happen in graphs. See NanLogger for details.
func (*NanLogger) AttachToExec ¶ added in v0.8.0
func (l *NanLogger) AttachToExec(exec ExecWithLogger)
AttachToExec will set the NanLogger as the default logger in exec. NanLogger acts as a pass-through logger, anything that is not marked as nanlogger.UniqueMessageId is passed through to whatever was the previous logger configured in exec.
A nil NanLogger is valid, and it will simply be a no-op.
func (*NanLogger) AttachToTrainer ¶ added in v0.8.0
AttachToTrainer makes sure that the logger is attached to every graph created by the trainer.
A nil NanLogger is valid, and it will simply be a no-op.
func (*NanLogger) PopScope ¶
func (l *NanLogger) PopScope()
PopScope removes the last entry in the current scope stack. These values are added by default to any new Trace.
A nil NanLogger is valid, and it will simply be a no-op.
func (*NanLogger) PushScope ¶
PushScope to current scope stack. These values are added by default to any new Trace.
A nil NanLogger is valid, and it will simply be a no-op.
func (*NanLogger) SetHandler ¶
SetHandler sets the function called when a `NaN` is observed. The default is DefaultHandler that prints out all information on the node and exits.
func (*NanLogger) Trace ¶
Trace the given node. This means the node is monitored and whenever a NaN is observed, the trace is printed and the program exit. Alternatively, a handler is called, see SetHandler.
A user-provided extra scope can be given: it's appended to the current NanLogger scope.
A nil NanLogger is valid, and it will simply be a no-op.
type Trace ¶
type Trace struct {
// StackTrace of where the monitored node was created, stored as an error that can be printed.
StackTrace error
// Scope saved when monitor node was created.
Scope []string
}
Trace information of a node that is set to monitor. This is what printed out when a `NaN` is found, or passed to a handler function, if one is set.
Source Files