Documentation ¶
Overview ¶
Package trc provides in-process request tracing, an efficient alternative to logging. The package is inspired by https://golang.org/x/net/trace, much gratitude to those authors.
The basic idea is that applications should log not by sending events to a destination like stdout or a file on disk, but instead by adding events to a value retrieved from the context, known as a Trace.
Traces are created for each operation processed by your application, e.g. every incoming HTTP request. Each trace is given a semantically meaningful category, and injected into the downstream context so applications can add events over the course of the operation. The trace is marked as finished when the operation completes.
Collector collects traces into per-category ring buffers. Collected traces can be queried over HTTP via [trcweb.NewServer]. That interface is fairly rich, allowing traces to be selected by category, minimum duration, successful vs. errored, and so on.
There are a few caveats. This approach is only suitable for applications that do their work in the context of a trace-related operation, and which reliably have access to a context value. Only the most recent traces are maintained, so long term historical data is not available. And, becase traces are maintained in memory, if a process crashes or restarts, all previous data is by default lost.
Even with these caveats, in-process request tracing often provides a better user experience than traditional logging. The value of application telemetry tends to be highly correlated to age. A rich interface over just the most recent data can be surprisingly powerful.
Most applications should not import this package directly, and should instead use github.com/peterbourgon/trc/eztrc, which provides an API specifically designed for common use cases.
Index ¶
- Constants
- Variables
- func SetTraceMaxEvents(n int)
- func SetTraceStacks(enable bool)
- type Broker
- type CategoryStats
- type Collector
- func (c *Collector) NewTrace(ctx context.Context, category string) (context.Context, Trace)
- func (c *Collector) Search(ctx context.Context, req *SearchRequest) (*SearchResponse, error)
- func (c *Collector) SetCategorySize(cap int) *Collector
- func (c *Collector) SetDecorators(decorators ...DecoratorFunc) *Collector
- func (c *Collector) SetNewTrace(newTrace NewTraceFunc) *Collector
- func (c *Collector) SetSourceName(name string) *Collector
- func (c *Collector) Stream(ctx context.Context, f Filter, ch chan<- Trace) (StreamStats, error)
- func (c *Collector) StreamStats(ctx context.Context, ch chan<- Trace) (StreamStats, error)
- type CollectorConfig
- type DecoratorFunc
- type Event
- type Filter
- type Frame
- type MultiSearcher
- type NewTraceFunc
- type SearchRequest
- type SearchResponse
- type SearchStats
- type Searcher
- type StaticTrace
- func (st *StaticTrace) Category() string
- func (st *StaticTrace) Duration() time.Duration
- func (st *StaticTrace) Errored() bool
- func (st *StaticTrace) Errorf(format string, args ...any)
- func (st *StaticTrace) Events() []Event
- func (st *StaticTrace) Finish()
- func (st *StaticTrace) Finished() bool
- func (st *StaticTrace) ID() string
- func (st *StaticTrace) LazyErrorf(format string, args ...any)
- func (st *StaticTrace) LazyTracef(format string, args ...any)
- func (st *StaticTrace) Source() string
- func (st *StaticTrace) Started() time.Time
- func (st *StaticTrace) Tracef(format string, args ...any)
- func (st *StaticTrace) TrimStacks(depth int) *StaticTrace
- type StreamStats
- type Trace
- func Get(ctx context.Context) Trace
- func MaybeGet(ctx context.Context) (Trace, bool)
- func New(ctx context.Context, source, category string, decorators ...DecoratorFunc) (context.Context, Trace)
- func Prefix(ctx context.Context, format string, args ...any) (context.Context, Trace)
- func Put(ctx context.Context, tr Trace) (context.Context, Trace)
- func Region(ctx context.Context, name string) (context.Context, Trace, func())
- func SetMaxEvents(tr Trace, maxEvents int) (Trace, bool)
Constants ¶
const ( // SearchLimitMin is the minimum search limit. SearchLimitMin = 1 // SearchLimitDefault is the default search limit. SearchLimitDefault = 10 // SearchLimitMax is the maximum search limit. SearchLimitMax = 250 )
Variables ¶
var DefaultBucketing = []time.Duration{ 0, 100 * time.Microsecond, 1 * time.Millisecond, 5 * time.Millisecond, 10 * time.Millisecond, 25 * time.Millisecond, 50 * time.Millisecond, 100 * time.Millisecond, 1000 * time.Millisecond, }
DefaultBucketing is the default set of time buckets used in search stats.
Functions ¶
func SetTraceMaxEvents ¶ added in v0.0.3
func SetTraceMaxEvents(n int)
SetTraceMaxEvents sets the max number of events that will be stored in a core trace. Once a core trace has the maximum number of events, additional events increment a "truncated" counter, which is represented as a single final event. The default is 1000, the minimum is 10, and the maximum is 10000.
Changing this value does not affect traces that have already been created.
func SetTraceStacks ¶ added in v0.0.3
func SetTraceStacks(enable bool)
SetTraceStacks sets a boolean that determines whether trace events include stack traces. By default, trace event stacks are enabled, because they're generally very useful. However, computing stack traces can be the single most computationally heavy part of using package trc, so disabling them altogether can be a significant performance optimization.
Changing this value does not affect traces that have already been created.
Types ¶
type Broker ¶ added in v0.0.3
type Broker struct {
// contains filtered or unexported fields
}
Broker allows traces to be published to a set of subscribers.
func (*Broker) Publish ¶ added in v0.0.3
Publish the trace, transformed via NewStreamTrace, to any active and matching subscribers. Sends to subscribers don't block and will drop.
func (*Broker) Stream ¶ added in v0.0.3
Stream will forward a copy of every trace created in the collector matching the filter to the provided channel. If the channel is full, traces will be dropped. For reasons of efficiency, streamed trace events don't have stacks. Stream blocks until the context is canceled.
Note that if the filter has IsActive true, the caller will receive not only complete matching traces as they are finished, but also a single-event trace for each individual matching event as they are created. This can be an enormous volume of data, please be careful.
func (*Broker) StreamStats ¶ added in v0.0.3
StreamStats returns statistics about a currently active subscription.
type CategoryStats ¶ added in v0.0.3
type CategoryStats struct { Category string `json:"category"` EventCount int `json:"event_count"` ActiveCount int `json:"active_count"` BucketCounts []int `json:"bucket_counts"` ErroredCount int `json:"errored_count"` Oldest time.Time `json:"oldest"` Newest time.Time `json:"newest"` // contains filtered or unexported fields }
CategoryStats represents statistics for all traces in a specific category.
func NewCategoryStats ¶ added in v0.0.3
func NewCategoryStats(category string, bucketing []time.Duration) *CategoryStats
NewCategoryStats returns an empty category stats for the given category, and with the given bucketing.
func (*CategoryStats) EventRate ¶ added in v0.0.3
func (cs *CategoryStats) EventRate() (r float64)
EventRate is an approximate measure of events per second in this category.
func (*CategoryStats) IsZero ¶ added in v0.0.3
func (cs *CategoryStats) IsZero() bool
IsZero returns true if the stats are not properly initialized or empty.
func (*CategoryStats) Merge ¶ added in v0.0.3
func (cs *CategoryStats) Merge(other *CategoryStats)
Merge the other category stats into this one.
func (*CategoryStats) TotalCount ¶ added in v0.0.3
func (cs *CategoryStats) TotalCount() int
TotalCount returns the total number of traces in the category.
func (*CategoryStats) TraceRate ¶ added in v0.0.3
func (cs *CategoryStats) TraceRate() (r float64)
TraceRate is an approximate measure of traces per second in this category.
type Collector ¶ added in v0.0.3
type Collector struct {
// contains filtered or unexported fields
}
Collector maintains a set of traces in memory, grouped by category.
func NewCollector ¶ added in v0.0.3
func NewCollector(cfg CollectorConfig) *Collector
NewCollector returns a new collector with the provided config.
func NewDefaultCollector ¶ added in v0.0.3
func NewDefaultCollector() *Collector
NewDefaultCollector returns a new collector with the source "default" and using New to produce new traces.
func (*Collector) NewTrace ¶ added in v0.0.3
NewTrace produces a new trace in the collector with the given category, injects it into the given context, and returns a new derived context containing the trace, as well as the new trace itself.
func (*Collector) Search ¶ added in v0.0.3
func (c *Collector) Search(ctx context.Context, req *SearchRequest) (*SearchResponse, error)
Search the collector for traces, according to the provided search request.
func (*Collector) SetCategorySize ¶ added in v0.0.3
SetCategorySize resets the max size of each category in the collector. If any categories are currently larger than the given capacity, they will be reduced by dropping old traces. The default capacity is 1000.
The method returns its receiver to allow for builder-style construction.
func (*Collector) SetDecorators ¶ added in v0.0.3
func (c *Collector) SetDecorators(decorators ...DecoratorFunc) *Collector
SetDecorators completely resets the decorators used by the collector.
The method returns its receiver to allow for builder-style construction.
func (*Collector) SetNewTrace ¶ added in v0.0.3
func (c *Collector) SetNewTrace(newTrace NewTraceFunc) *Collector
SetNewTrace sets the new trace function used by the collector.
The method returns its receiver to allow for builder-style construction.
func (*Collector) SetSourceName ¶ added in v0.0.3
SetSourceName sets the source used by the collector.
The method returns its receiver to allow for builder-style construction.
func (*Collector) Stream ¶ added in v0.0.3
Stream traces matching the filter to the channel, returning when the context is canceled. See Broker.Stream for more details.
func (*Collector) StreamStats ¶ added in v0.0.3
StreamStats returns statistics about a currently active subscription.
type CollectorConfig ¶ added in v0.0.3
type CollectorConfig struct { // Source is used as the source for all traces created within the collector. // If not provided, the "default" source is used. Source string // NewTrace is used to construct the traces in the collector. If not // provided, the [New] function is used. NewTrace NewTraceFunc // Decorators are applied to every new trace created in the collector. Decorators []DecoratorFunc // Broker is used for streaming traces and events. If not provided, a new // broker will be constructed and used. Broker *Broker }
CollectorConfig captures the configuration parameters for a collector.
type DecoratorFunc ¶ added in v0.0.3
DecoratorFunc is a function that decorates a trace in some way. It's similar to an HTTP middleware. Decorators can be provided to a Collector and will be applied to every trace created in that collector.
func LogDecorator ¶ added in v0.0.3
func LogDecorator(dst io.Writer) DecoratorFunc
LogDecorator logs a simple string to the provided destination when the trace is created, on every event, and when the trace is finished. The logged string is a reduced form of the full trace, containing only the trace ID and the single event that triggered the log.
func LoggerDecorator ¶ added in v0.0.3
func LoggerDecorator(logger *log.Logger) DecoratorFunc
LoggerDecorator is like LogDecorator, but uses a log.Logger.
type Event ¶
type Event struct { When time.Time `json:"when"` What string `json:"what"` Stack []Frame `json:"stack,omitempty"` IsError bool `json:"is_error,omitempty"` }
Event is a traced event, similar to a log event, which is created in the context of a specific trace, via methods like Tracef.
type Filter ¶ added in v0.0.3
type Filter struct { Sources []string `json:"sources,omitempty"` IDs []string `json:"ids,omitempty"` Category string `json:"category,omitempty"` IsActive bool `json:"is_active,omitempty"` IsFinished bool `json:"is_finished,omitempty"` MinDuration *time.Duration `json:"min_duration,omitempty"` IsSuccess bool `json:"is_success,omitempty"` IsErrored bool `json:"is_errored,omitempty"` Query string `json:"query,omitempty"` // contains filtered or unexported fields }
Filter is a set of rules that can be applied to an individual trace, which will either be allowed (pass) or rejected (fail).
func (*Filter) Allow ¶ added in v0.0.3
Allow returns true if the provided trace satisfies all of the conditions in the filter.
type Frame ¶
Frame is a single call frame in an event's call stack.
func (Frame) CompactFileLine ¶
CompactFileLine returns a human-readable representation of the file and line, intended to be used in user-facing interfaces.
type MultiSearcher ¶ added in v0.0.3
type MultiSearcher []Searcher
MultiSearcher allows multiple searchers to be searched as one.
func (MultiSearcher) Search ¶ added in v0.0.3
func (ms MultiSearcher) Search(ctx context.Context, req *SearchRequest) (*SearchResponse, error)
Search scatters the request over the searchers, gathers responses, and merges them into a single response returned to the caller.
type NewTraceFunc ¶ added in v0.0.3
type NewTraceFunc func(ctx context.Context, source string, category string, decorators ...DecoratorFunc) (context.Context, Trace)
NewTraceFunc describes a function that produces a new trace with a specific source and category, and which is decorated by the given decorators. It returns a context containing the new trace, as well as the new trace itself.
type SearchRequest ¶ added in v0.0.3
type SearchRequest struct { Bucketing []time.Duration `json:"bucketing,omitempty"` Filter Filter `json:"filter,omitempty"` Limit int `json:"limit,omitempty"` StackDepth int `json:"stack_depth,omitempty"` // 0 is default stacks, -1 for no stacks }
SearchRequest describes a complete search request.
func (*SearchRequest) Normalize ¶ added in v0.0.3
func (req *SearchRequest) Normalize() []error
Normalize ensures the search request is valid, modifying it if necessary. It returns any errors encountered in the process.
func (SearchRequest) String ¶ added in v0.0.3
func (req SearchRequest) String() string
String implements fmt.Stringer.
type SearchResponse ¶ added in v0.0.3
type SearchResponse struct { Request *SearchRequest `json:"request,omitempty"` Sources []string `json:"sources"` TotalCount int `json:"total_count"` MatchCount int `json:"match_count"` Traces []*StaticTrace `json:"traces"` Stats *SearchStats `json:"stats,omitempty"` Problems []string `json:"problems,omitempty"` Duration time.Duration `json:"duration"` }
SearchResponse returned by a search request.
type SearchStats ¶ added in v0.0.3
type SearchStats struct { Bucketing []time.Duration `json:"bucketing"` Categories map[string]*CategoryStats `json:"categories"` }
SearchStats are statistics over the complete set of traces that were queried as part of a search request.
func NewSearchStats ¶ added in v0.0.3
func NewSearchStats(bucketing []time.Duration) *SearchStats
NewSearchStats creates a new and empty search stats, with the given time buckets for grouping finished traces.
func (*SearchStats) AllCategories ¶ added in v0.0.3
func (ss *SearchStats) AllCategories() []*CategoryStats
AllCategories returns category stats for all known categories, as well as the synthetic Overall category.
func (*SearchStats) IsZero ¶ added in v0.0.3
func (ss *SearchStats) IsZero() bool
IsZero returns true if the stats are empty.
func (*SearchStats) Merge ¶ added in v0.0.3
func (ss *SearchStats) Merge(other *SearchStats)
Merge the other stats into this one.
func (*SearchStats) Observe ¶ added in v0.0.3
func (ss *SearchStats) Observe(trs ...Trace)
Observe the given traces into the search stats.
func (*SearchStats) Overall ¶ added in v0.0.3
func (ss *SearchStats) Overall() *CategoryStats
Overall returns a synthetic category stats representing all categories.
type Searcher ¶ added in v0.0.3
type Searcher interface {
Search(context.Context, *SearchRequest) (*SearchResponse, error)
}
Searcher models anything that can serve search requests.
type StaticTrace ¶ added in v0.0.3
type StaticTrace struct { TraceSource string `json:"source"` TraceID string `json:"id"` TraceCategory string `json:"category"` TraceStarted time.Time `json:"started"` TraceDuration time.Duration `json:"duration"` TraceDurationStr string `json:"duration_str,omitempty"` TraceDurationSec float64 `json:"duration_sec,omitempty"` TraceFinished bool `json:"finished,omitempty"` TraceErrored bool `json:"errored,omitempty"` TraceEvents []Event `json:"events,omitempty"` }
StaticTrace is a "snapshot" of a trace which can be sent over the wire.
func NewSearchTrace ¶ added in v0.0.3
func NewSearchTrace(tr Trace) *StaticTrace
NewSearchTrace produces a static trace intended for a search response.
func NewStreamTrace ¶ added in v0.0.3
func NewStreamTrace(tr Trace) *StaticTrace
NewStreamTrace produces a static trace meant for streaming. If the trace is active, only the most recent event is included. Also, stacks are removed from every event.
func (*StaticTrace) Category ¶ added in v0.0.3
func (st *StaticTrace) Category() string
Category implements the Trace interface.
func (*StaticTrace) Duration ¶ added in v0.0.3
func (st *StaticTrace) Duration() time.Duration
Duration implements the Trace interface.
func (*StaticTrace) Errored ¶ added in v0.0.3
func (st *StaticTrace) Errored() bool
Errored implements the Trace interface.
func (*StaticTrace) Errorf ¶ added in v0.0.3
func (st *StaticTrace) Errorf(format string, args ...any)
Errorf implements the Trace interface.
func (*StaticTrace) Events ¶ added in v0.0.3
func (st *StaticTrace) Events() []Event
Events implements the Trace interface.
func (*StaticTrace) Finish ¶ added in v0.0.3
func (st *StaticTrace) Finish()
Finish implements the Trace interface.
func (*StaticTrace) Finished ¶ added in v0.0.3
func (st *StaticTrace) Finished() bool
Finished implements the Trace interface.
func (*StaticTrace) ID ¶ added in v0.0.3
func (st *StaticTrace) ID() string
ID implements the Trace interface.
func (*StaticTrace) LazyErrorf ¶ added in v0.0.3
func (st *StaticTrace) LazyErrorf(format string, args ...any)
LazyErrorf implements the Trace interface.
func (*StaticTrace) LazyTracef ¶ added in v0.0.3
func (st *StaticTrace) LazyTracef(format string, args ...any)
LazyTracef implements the Trace interface.
func (*StaticTrace) Source ¶ added in v0.0.3
func (st *StaticTrace) Source() string
Source implements the Trace interface.
func (*StaticTrace) Started ¶ added in v0.0.3
func (st *StaticTrace) Started() time.Time
Started implements the Trace interface.
func (*StaticTrace) Tracef ¶ added in v0.0.3
func (st *StaticTrace) Tracef(format string, args ...any)
Tracef implements the Trace interface.
func (*StaticTrace) TrimStacks ¶ added in v0.0.3
func (st *StaticTrace) TrimStacks(depth int) *StaticTrace
TrimStacks reduces the stacks of every event in the trace based on depth. A depth of 0 means "no change" -- to remove stacks, use a depth of -1.
type StreamStats ¶ added in v0.0.3
type StreamStats struct { // Skips is how many traces were considered but didn't pass the filter. Skips int `json:"skips"` // Sends is how many traces were successfully sent to the subscriber. Sends int `json:"sends"` // Drops is how many traces were dropped due to lack of capacity. Drops int `json:"drops"` }
StreamStats is metadata about a currently active subscription.
func (StreamStats) String ¶ added in v0.0.3
func (s StreamStats) String() string
String implements fmt.Stringer.
type Trace ¶
type Trace interface { // ID returns an identifier for the trace which should be automatically // generated during construction, and should be unique within a given // instance. ID() string // Source returns a human-readable string representing the origin of the // trace, which is typically the instance of the program where the trace was // constructed. Source() string // Category returns the category of the trace, which should be provided by // the caller when the trace is created. Category() string // Started returns when the trace was created, preferably in UTC. Started() time.Time // Duration returns how long the trace is (or was) active, which is the time // between the started timestamp and when the trace was finished. If the // trace is still active, it returns the time since the started timestamp. Duration() time.Duration // Tracef adds a normal event to the trace, with the given format string and // args. Args are evaluated immediately. Tracef(format string, args ...any) // LazyTracef adds a normal event to the trace, with the given format string // and args. Args are stored in their raw form and evaulated lazily, when // the event is first read. Callers should be very careful to ensure that // args passed to this method will remain valid indefinitely. LazyTracef(format string, args ...any) // Errorf adds an error event to the trace, with the given format string and // args. It marks the trace as errored. Args are evaluated immediately. Errorf(format string, args ...any) // LazyErrorf adds an error event to the trace, with the given format string // and args. It marks the trace as errored. Args are stored in their raw // form and evaulated lazily, when the event is first read. Callers should // be very careful to ensure that args passed to this method will remain // valid indefinitely. LazyErrorf(format string, args ...any) // Finish marks the trace as finished. Once finished, a trace is "frozen", // and any method that would modify the trace becomes a no-op. Finish() // Finished returns true if Finish has been called. Finished() bool // Errored returns true if Errorf or LazyErrorf has been called. Errored() bool // Events returns all of the events collected by the trace, oldest to // newest. Events are produced by Tracef, LazyTracef, Errorf, and // LazyErrorf. Events() []Event }
Trace is a collection of metadata and events for a single operation, typically a request, in a program. Traces are normally accessed through a context, and maintained in a Collector.
New produces a default implementation of a Trace which is suitable for most use cases. Consumers can extend that implementation via DecoratorFunc, or provide their own implementation entirely. Trace implementations must be safe for concurrent use.
Note that traces are typically created for every operation, but are accessed only upon explicit request, for example when an operator is diagnosing a problem. Consequently, traces are written far more often than they are read. Implementations should keep this access pattern in mind, and optimize for writes rather than reads.
Trace implementations may optionally implement SetMaxEvents(int), to allow callers to modify the maximum number of events that will be stored in the trace. This method, if it exists, is called by SetMaxEvents.
Trace implementations may optionally implement Free(), to release any resources claimed by the trace to an e.g. sync.Pool. This method, if it exists, is called by the Collector when a trace is dropped.
func Get ¶
Get the trace from the context, if it exists. If not, an "orphan" trace is created and returned (but not injected into the context).
func MaybeGet ¶
MaybeGet returns the trace in the context, if it exists. If not, MaybeGet returns a nil trace and false.
func New ¶
func New(ctx context.Context, source, category string, decorators ...DecoratorFunc) (context.Context, Trace)
New creates a new core trace with the given source and category, and injects it into the given context. It returns a new context containing that trace, and the trace itself.
func Prefix ¶
Prefix decorates the trace in the context such that every trace event will be prefixed with the string specified by format and args. Those args are not evaluated when Prefix is called, but are instead prefixed to the format and args of trace events made against the returned trace.
func Put ¶
Put the given trace into the context, and return a new context containing that trace, as well as the trace itself. If the context already contained a trace, it becomes "shadowed" by the new trace.
func Region ¶
Region provides more detailed tracing of regions of code, usually functions, which is visible in the trace event "what" text. It decorates the trace in the context by annotating events with the provided name, and also creates a standard library runtime/trace.Region with the same name.
Typical usage is as follows.
func foo(ctx context.Context, id int) { ctx, tr, finish := trc.Region(ctx, "foo") defer finish() ... }
This produces hierarchical trace events as follows.
→ foo · trace event in foo · another event in foo · → bar · · something in bar · ← bar [1.23ms] · final event in foo ← foo [2.34ms]
Region can significantly impact performance. Use it sparingly.
func SetMaxEvents ¶
SetMaxEvents tries to set the max events for a specific trace, by checking if the trace implements the method SetMaxEvents(int), and, if so, calling that method with the given max events value. Returns the given trace, and a boolean representing whether or not the call was successful.
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
cmd
|
|
trc
trc is a CLI tool for interacting with trc web servers.
|
trc is a CLI tool for interacting with trc web servers. |
Package eztrc provides an easy-to-use API for common tracing use cases.
|
Package eztrc provides an easy-to-use API for common tracing use cases. |
internal
|
|
trcdebug
Package trcdebug contains debug information for the trc package.
|
Package trcdebug contains debug information for the trc package. |
trcringbuf
Package trcringbuf provides a fixed-size ring buffer for traces.
|
Package trcringbuf provides a fixed-size ring buffer for traces. |
trcutil
Package trcutil contains simple helper functions used throughout the module.
|
Package trcutil contains simple helper functions used throughout the module. |
Package trcweb provides an HTTP interface to traces.
|
Package trcweb provides an HTTP interface to traces. |
assets
Package assets contains assets for the trc web interface.
|
Package assets contains assets for the trc web interface. |