trace

Documentation

Overview

Package trace wraps standard tracing for outreach.

This package wraps honeycomb tracing

Trace Initialization

Applications should call `trace.StartTracing(serviceName)` and `trace.StopTracing()` in their `main` like so:

func main() {
     trace.StartTracing("example")
     defer trace.StopTracing()

     ... main app logic ...
}

See https://github.com/getoutreach/gobox/blob/master/cmd/example/main.go.

Servers and incoming requests

The httpx/pkg/handlers package wraps the required trace header parsing logic and so applications that use `handlers.Endpoint` do not have to do anything special here.

Custom http servers should wrap their request handling code like so:

r = r.WithContext(trace.ContextFromHTTP(r, "my endpoint"))
defer trace.End(r.Context()
... do actual request handling ...

Non-HTTP servers should wrap their request handling like so:

ctx = trace.StartTrace(ctx, "my endpoint")
defer trace.End(ctx)
... do actual request handling ...

Clients

Propagating trace headers to HTTP clients or non-HTTP clients is not yet implemented here. Please see ETC-190.

Tracing calls

Any interesting function (such as model fetches or redis fetches) should use the following pattern:

func MyInterestingRedisFunction(ctx context.Context, ...) error {
    ctx = trace.StartCall(ctx, "redis", RedisInfo{...})
    defer trace.EndCall(ctx)

    .... actual work ...
    trace.AddInfo(ctx, xyzInfo)

    return trace.SetCallStatus(ctx, err)
}

This automatically updates metrics ("call_request_secconds" is the counter with "redis" as the name label), writes to debug/error logs and also writes traces to our tracing infrastructure

Trace calls can be nested.

Creating spans

Spans should rarely be needed but are available for when the metrics or default logging is not sufficient.

Spans are automatically considered `children` of the current trace or span (based on the `context`). The redis example above would look like so:

ctx = trace.StartTrace(ctx, "redis")
defer trace.End(ctx)
.... do actual redis call...

Adding tags

Tags can be added to the `current` span (or trace or call) by simply calling `trace.AddInfo`. Note that this accepts the same types that logging accepts. For instance, to record an error with redis:

result, err := redis.Call(....)
if err != nil {
   // if you are using trace.Call, then do trace.SetCallStatus
   // instead.
   trace.AddInfo(ctx, events.NewErrorInfo(err))
}

Index

• Constants
• func AddInfo(ctx context.Context, args ...log.Marshaler)
• func AddSpanInfo(ctx context.Context, args ...log.Marshaler)
• func AsGRPCCall() call.Option
• func AsHTTPCall() call.Option
• func AsOutboundCall() call.Option
• func CloseTracer(ctx context.Context)
• func ContextFromHTTP(r *http.Request, name string) context.Context
• func End(ctx context.Context)
• func EndCall(ctx context.Context)
• func EndTracing()
• func ForceSampleRate(ctx context.Context, rate uint) context.Context
• func ForceTracing(ctx context.Context) context.Context
• func FromHeaders(ctx context.Context, hdrs map[string][]string, name string) context.Context
• func FromHeadersAsync(ctx context.Context, hdrs map[string][]string, name string) context.Context
• func ID(ctx context.Context) string
• func IDs(ctx context.Context) log.Marshaler
• func InitTracer(ctx context.Context, serviceName string) error
• func NewTransport(old http.RoundTripper) http.RoundTripper
• func SetCallError(ctx context.Context, err error) error
• func SetCallStatus(ctx context.Context, err error) error
• func SetCallTypeGRPC(ctx context.Context) context.Context
• func SetCallTypeHTTP(ctx context.Context) context.Context
• func SetCallTypeOutbound(ctx context.Context) context.Context
• func SetPresendHook(hook func(map[string]interface{}))
• func StartCall(ctx context.Context, cType string, args ...log.Marshaler) context.Context
• func StartSpan(ctx context.Context, name string, args ...log.Marshaler) context.Context
• func StartSpanAsync(ctx context.Context, name string, args ...log.Marshaler) context.Context
• func StartTrace(ctx context.Context, name string) context.Context
• func StartTracing(serviceName string) error
• func ToHeaders(ctx context.Context) map[string][]string
• func WithScheduledTime(t time.Time) call.Option
• type Config
  • func (c *Config) Load() error
• type GlobalTags
  • func (g *GlobalTags) MarshalLog(addField func(key string, v interface{}))
• type Honeycomb

Constants

const (
	// Header that enforces the tracing for particular request
	HeaderForceTracing = "X-Force-Trace"
)

Functions

func AddInfo

func AddInfo(ctx context.Context, args ...log.Marshaler)

AddInfo updates the current span with the provided fields. If a call exists, it updates the call info args with the passed in log marshalers

This is not propagated to child spans automatically.

It does nothing if there isn't a current span.

func AddSpanInfo

func AddSpanInfo(ctx context.Context, args ...log.Marshaler)

AddSpanInfo updates the current span with the provided fields.

It does nothing if there isn't a current span.

func AsGRPCCall

func AsGRPCCall() call.Option

AsGRPCCall set the call type to GRPC

func AsHTTPCall

func AsHTTPCall() call.Option

AsHTTPCall set the call type to HTTP

func AsOutboundCall

func AsOutboundCall() call.Option

AsOutboundCall set the call type to Outbound

func CloseTracer

func CloseTracer(ctx context.Context)

CloseTracer stops all tracing and sends any queued traces.

This should be called when an application is exiting, or when you want to terminate the tracer.

func ContextFromHTTP

func ContextFromHTTP(r *http.Request, name string) context.Context

ContextFromHTTP starts a new trace from an incoming http request.

Use trace.End to end this.

func End

func End(ctx context.Context)

End ends a span (or a trace started via StartTrace or ContextFromHTTP).

func EndCall

func EndCall(ctx context.Context)

EndCall calculates the duration of the call, writes to metrics, standard logs and closes the trace span.

This call should always be right after the StartCall in a defer pattern. See StartCall for the right pattern of usage.

EndCall, when called within a defer, catches any panics and rethrows them. Any panics are converted to errors and cause error logging to happen (as do any SetCallStatus calls)

func EndTracing

func EndTracing()

Deprecated: Use CloseTracer() instead. EndTracing stops the tracing infrastructure.

This should be called at the exit of the application.

func ForceSampleRate

func ForceSampleRate(ctx context.Context, rate uint) context.Context

ForceSampleRate will force a desired sample rate for the given trace and all children of said trace. The sample rate in practice will be 1/<rate>.

For example, if you invoked:

ctx = trace.ForceSampleRate(ctx, 1000)

The trace spawned from that and all of it's children would be sampled at a rate of 1/1000, or 1/10 of a percent (.1%).

func ForceTracing

func ForceTracing(ctx context.Context) context.Context

ForceTracing will enforce tracing for processing started with returned context and all downstream services that will be invoken on the way.

func FromHeaders

func FromHeaders(ctx context.Context, hdrs map[string][]string, name string) context.Context

FromHeaders fetches trace info from a headers map

func FromHeadersAsync

func FromHeadersAsync(ctx context.Context, hdrs map[string][]string, name string) context.Context

FromHeadersAsync fetches trace info from a headers map and kicks off an async trace.

func ID

func ID(ctx context.Context) string

ID returns an ID for use with external services to propagate tracing context. The ID returned will be the honeycomb trace ID (if honeycomb is enabled) or an empty string if neither are enabled.

func IDs

func IDs(ctx context.Context) log.Marshaler

IDs returns a log-compatible tracing scope (IDs) data built from the context suitable for logging.

func InitTracer

func InitTracer(ctx context.Context, serviceName string) error

InitTracer starts all tracing infrastructure.

This needs to be called before sending any traces otherwise they will not be published.

func NewTransport

func NewTransport(old http.RoundTripper) http.RoundTripper

NewTransport creates a new transport which propagates the current trace context.

Usage:

client := &http.Client{Transport: trace.NewTransport(nil)}
resp, err := client.Get("/ping")

For most cases, use the httpx/pkg/fetch package as it also logs the request, updates latency metrics and adds traces with full info

Note: the request context must be derived from StartSpan/StartTrace etc.

func SetCallError

func SetCallError(ctx context.Context, err error) error

SetCallError is deprecated and will directly call into SetCallStatus for backward compatibility

func SetCallStatus

func SetCallStatus(ctx context.Context, err error) error

SetCallStatus can be optionally called to set status of the call. When the error occurs on the current call, the error will be traced. When the error is nil, no-op from this function

func SetCallTypeGRPC

func SetCallTypeGRPC(ctx context.Context) context.Context

Deprecated: use AsGrpcCall call.Option instead SetTypeGRPC is meant to set the call type to GRPC on a context that has already been initialized for tracing via StartCall or StartExternalCall.

func SetCallTypeHTTP

func SetCallTypeHTTP(ctx context.Context) context.Context

Deprecated: use AsHttpCall call.Option instead SetTypeHTTP is meant to set the call type to HTTP on a context that has already been initialized for tracing via StartCall or StartExternalCall.

func SetCallTypeOutbound

func SetCallTypeOutbound(ctx context.Context) context.Context

Deprecated: use AsOutboundCall call.Option instead SetCallTypeOutbound is meant to set the call type to Outbound on a context that has already been initialized for tracing via StartCall or StartExternalCall.

func SetPresendHook

func SetPresendHook(hook func(map[string]interface{}))

SetPresendHook sets the honeycomb presend hook for testing

func StartCall

func StartCall(ctx context.Context, cType string, args ...log.Marshaler) context.Context

StartCall is used to start an internal call. For external calls please use StartExternalCall.

This takes care of standard logging, metrics and tracing for "calls"

Typical usage:

ctx = trace.StartCall(ctx, "sql", SQLEvent{Query: ...})
defer trace.EndCall(ctx)

return trace.SetCallStatus(ctx, sqlCall(...));

The callType should be broad category (such as "sql", "redis" etc) as as these are used for metrics and cardinality issues come into play. Do note that commonly used call types are exported as constants in this package and should be used whenever possible. The most common call types are http (trace.CallTypeHTTP) and grpc (trace.CallTypeGRPC).

Use the extra args to add stuff to traces and logs and these can have more information as needed (including actual queries for instance).

The log includes a initial Debug entry and a final Error entry if the call failed (but no IDs entry if the call succeeded). Success or failure is determined by whether there was a SetCallStatus or not. (Panics detected in EndCall are considered errors).

StartCalls can be nested.

func StartSpan

func StartSpan(ctx context.Context, name string, args ...log.Marshaler) context.Context

StartSpan starts a new span.

Use trace.End to end this.

func StartSpanAsync

func StartSpanAsync(ctx context.Context, name string, args ...log.Marshaler) context.Context

StartSpanAsync starts a new async span.

An async span does not have to complete before the parent span completes.

Use trace.End to end this.

func StartTrace

func StartTrace(ctx context.Context, name string) context.Context

StartTrace starts a new root span/trace.

Use trace.End to end this.

func StartTracing

func StartTracing(serviceName string) error

Deprecated: Use InitTracer() instead. StartTracing starts the tracing infrastructure.

This should be called at the start of the application.

func ToHeaders

func ToHeaders(ctx context.Context) map[string][]string

ToHeaders writes the current trace context into a headers map

func WithScheduledTime

func WithScheduledTime(t time.Time) call.Option

WithScheduledTime set the call Info scheduled at time

Types

type Config

type Config struct {
	Honeycomb  `yaml:"Honeycomb"`
	GlobalTags `yaml:"GlobalTags,omitempty"`
}

tracing config goes into trace.yaml

func (*Config) Load

func (c *Config) Load() error

type GlobalTags

type GlobalTags struct {
	DevEmail string `yaml:"DevEmail,omitempty"`
}

func (*GlobalTags) MarshalLog

func (g *GlobalTags) MarshalLog(addField func(key string, v interface{}))

type Honeycomb

type Honeycomb struct {
	Enabled       bool       `yaml:"Enabled"`
	APIHost       string     `yaml:"APIHost"`
	Dataset       string     `yaml:"Dataset"`
	SamplePercent float64    `yaml:"SamplePercent"`
	Debug         bool       `yaml:"Debug"`
	Stdout        bool       `yaml:"Stdout"`
	APIKey        cfg.Secret `yaml:"APIKey"`
}