trace

Documentation

Overview

Package trace wraps standard tracing for outreach.

This package wraps honeycomb tracing

Trace Initialization

Applications should call `trace.StartTracing(serviceName)` and `trace.StopTracing()` in their `main` like so:

func main() {
     trace.StartTracing("example")
     defer trace.StopTracing()

     ... main app logic ...
}

See https://github.com/getoutreach/gobox/blob/master/cmd/example/main.go.

Servers and incoming requests

The httpx/pkg/handlers package wraps the required trace header parsing logic and so applications that use `handlers.Endpoint` do not have to do anything special here.

Custom http servers should wrap their request handling code like so:

		trace.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) *roundtripperState {
		  trace.StartSpan(r.Context(), "my endpoint")
		  defer trace.End(r.Context())
		  ... do actual request handling ...
     }), "my endpoint")

Non-HTTP servers should wrap their request handling like so:

ctx = trace.StartSpan(ctx, "my endpoint")
defer trace.End(ctx)
... do actual request handling ...

Clients should use a Client with the provided transport like so:

ctx = trace.StartSpan(ctx, "my call")
defer trace.End(ctx)
client := http.Client{Transport: trace.NewTransport(nil)}
... do actual call using the new client ...

Tracing calls

Any interesting function (such as model fetches or redis fetches) should use the following pattern:

func MyInterestingRedisFunction(ctx context.Context, ...) error {
    ctx = trace.StartCall(ctx, "redis", RedisInfo{...})
    defer trace.EndCall(ctx)

    .... actual work ...
    trace.AddInfo(ctx, xyzInfo)

    return trace.SetCallStatus(ctx, err)
}

This automatically updates metrics ("call_request_secconds" is the counter with "redis" as the name label), writes to debug/error logs and also writes traces to our tracing infrastructure

Trace calls can be nested.

Creating spans

Spans should rarely be needed but are available for when the metrics or default logging is not sufficient.

Spans are automatically considered `children` of the current trace or span (based on the `context`). The redis example above would look like so:

ctx = trace.StartSpan(ctx, "redis")
defer trace.End(ctx)
.... do actual redis call...

Adding tags

Tags can be added to the `current` span (or trace or call) by simply calling `trace.AddInfo`. Note that this accepts the same types that logging accepts. For instance, to record an error with redis:

result, err := redis.Call(....)
if err != nil {
   // if you are using trace.Call, then do trace.SetCallStatus
   // instead.
   trace.AddInfo(ctx, events.NewErrorInfo(err))
}

Index

• Constants
• func AddInfo(ctx context.Context, args ...log.Marshaler)
• func AddSpanInfo(ctx context.Context, args ...log.Marshaler)
• func AsGRPCCall() call.Option
• func AsHTTPCall() call.Option
• func AsOutboundCall() call.Option
• func CloseTracer(ctx context.Context)
• func End(ctx context.Context)
• func EndCall(ctx context.Context)
• func EndTracing()
• func ForceTracing(ctx context.Context) context.Context
• func FromHeaders(ctx context.Context, hdrs map[string][]string, name string) context.Context
• func ID(ctx context.Context) string
• func IDs(ctx context.Context) log.Marshaler
• func InitTracer(ctx context.Context, serviceName string) error
• func NewHandler(handler http.Handler, operation string) http.Handler
• func NewTransport(old http.RoundTripper) http.RoundTripper
• func RegisterSpanProcessor(s sdktrace.SpanProcessor)
• func SetCallError(ctx context.Context, err error) error
• func SetCallStatus(ctx context.Context, err error) error
• func SetCallTypeGRPC(ctx context.Context) context.Context
• func SetCallTypeHTTP(ctx context.Context) context.Context
• func SetCallTypeOutbound(ctx context.Context) context.Context
• func SetSpanProcessorHook(hook func([]attribute.KeyValue))
• func StartCall(ctx context.Context, cType string, args ...log.Marshaler) context.Context
• func StartSpan(ctx context.Context, name string, args ...log.Marshaler) context.Context
• func StartSpanAsync(ctx context.Context, name string, args ...log.Marshaler) context.Context
• func StartTrace(ctx context.Context, name string) context.Context
• func StartTracing(serviceName string) error
• func ToHeaders(ctx context.Context) map[string][]string
• func WithScheduledTime(t time.Time) call.Option
• type Annotator
  • func (a Annotator) ForceFlush(context.Context) error
  • func (a Annotator) OnEnd(s sdktrace.ReadOnlySpan)
  • func (a Annotator) OnStart(_ context.Context, s sdktrace.ReadWriteSpan)
  • func (a Annotator) Shutdown(context.Context) error
• type Config
  • func (c *Config) Load() error
• type GlobalTags
  • func (g *GlobalTags) MarshalLog(addField func(key string, v interface{}))
• type Handler
  • func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)
• type Otel

Constants

const (
	// Header that enforces the tracing for particular request
	HeaderForceTracing = "X-Force-Trace"
	// Header used by OpenTelemetry to propagate traces
	OtelPropagationHeader = "Traceparent"
)

Functions

func AddInfo

func AddInfo(ctx context.Context, args ...log.Marshaler)

AddInfo updates the current span with the provided fields. If a call exists, it updates the call info args with the passed in log marshalers

This is not propagated to child spans automatically.

It does nothing if there isn't a current span.

func AddSpanInfo

func AddSpanInfo(ctx context.Context, args ...log.Marshaler)

AddSpanInfo updates the current span with the provided fields.

It does nothing if there isn't a current span.

func AsGRPCCall

func AsGRPCCall() call.Option

AsGRPCCall set the call type to GRPC

func AsHTTPCall

func AsHTTPCall() call.Option

AsHTTPCall set the call type to HTTP

func AsOutboundCall

func AsOutboundCall() call.Option

AsOutboundCall set the call type to Outbound

func CloseTracer

func CloseTracer(ctx context.Context)

CloseTracer stops all tracing and sends any queued traces.

This should be called when an application is exiting, or when you want to terminate the tracer.

func End

func End(ctx context.Context)

End ends a span (or a trace started via StartTrace).

func EndCall

func EndCall(ctx context.Context)

EndCall calculates the duration of the call, writes to metrics, standard logs and closes the trace span.

This call should always be right after the StartCall in a defer pattern. See StartCall for the right pattern of usage.

EndCall, when called within a defer, catches any panics and rethrows them. Any panics are converted to errors and cause error logging to happen (as do any SetCallStatus calls)

func EndTracing

func EndTracing()

Deprecated: Use CloseTracer() instead. EndTracing stops the tracing infrastructure.

This should be called at the exit of the application.

func ForceTracing

func ForceTracing(ctx context.Context) context.Context

ForceTracing will enforce tracing for processing started with returned context and all downstream services that will be invoken on the way.

func FromHeaders

func FromHeaders(ctx context.Context, hdrs map[string][]string, name string) context.Context

FromHeaders fetches trace info from a headers map

Only use for GRPC. Prefer NewHandler for http calls.

func ID

func ID(ctx context.Context) string

ID returns an ID for use with external services to propagate tracing context. The ID returned will be the honeycomb trace ID (if honeycomb is enabled) or an empty string if neither are enabled.

func IDs

func IDs(ctx context.Context) log.Marshaler

IDs returns a log-compatible tracing scope (IDs) data built from the context suitable for logging.

func InitTracer

func InitTracer(ctx context.Context, serviceName string) error

InitTracer starts all tracing infrastructure.

This needs to be called before sending any traces otherwise they will not be published.

func NewHandler

func NewHandler(handler http.Handler, operation string) http.Handler

NewHandler creates a new handlers which reads propagated headers from the current trace context.

Usage:

	  trace.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) *roundtripperState {
		trace.StartSpan(r.Context(), "my endpoint")
		defer trace.End(r.Context())
		... do actual request handling ...
   }), "my endpoint")

func NewTransport

func NewTransport(old http.RoundTripper) http.RoundTripper

NewTransport creates a new transport which propagates the current trace context.

Usage:

client := &http.Client{Transport: trace.NewTransport(nil)}
resp, err := client.Get("/ping")

For most cases, use the httpx/pkg/fetch package as it also logs the request, updates latency metrics and adds traces with full info

Note: the request context must be derived from StartSpan.

func RegisterSpanProcessor

func RegisterSpanProcessor(s sdktrace.SpanProcessor)

func SetCallError

func SetCallError(ctx context.Context, err error) error

SetCallError is deprecated and will directly call into SetCallStatus for backward compatibility

func SetCallStatus

func SetCallStatus(ctx context.Context, err error) error

SetCallStatus can be optionally called to set status of the call. When the error occurs on the current call, the error will be traced. When the error is nil, no-op from this function

func SetCallTypeGRPC

func SetCallTypeGRPC(ctx context.Context) context.Context

Deprecated: use AsGrpcCall call.Option instead SetTypeGRPC is meant to set the call type to GRPC on a context that has already been initialized for tracing via StartCall or StartExternalCall.

func SetCallTypeHTTP

func SetCallTypeHTTP(ctx context.Context) context.Context

Deprecated: use AsHttpCall call.Option instead SetTypeHTTP is meant to set the call type to HTTP on a context that has already been initialized for tracing via StartCall or StartExternalCall.

func SetCallTypeOutbound

func SetCallTypeOutbound(ctx context.Context) context.Context

Deprecated: use AsOutboundCall call.Option instead SetCallTypeOutbound is meant to set the call type to Outbound on a context that has already been initialized for tracing via StartCall or StartExternalCall.

func SetSpanProcessorHook

func SetSpanProcessorHook(hook func([]attribute.KeyValue))

SetSpanProcessorHook sets a hook to run when a span ends

func StartCall

func StartCall(ctx context.Context, cType string, args ...log.Marshaler) context.Context

StartCall is used to start an internal call. For external calls please use StartExternalCall.

This takes care of standard logging, metrics and tracing for "calls"

Typical usage:

ctx = trace.StartCall(ctx, "sql", SQLEvent{Query: ...})
defer trace.EndCall(ctx)

return trace.SetCallStatus(ctx, sqlCall(...));

The callType should be broad category (such as "sql", "redis" etc) as as these are used for metrics and cardinality issues come into play. Do note that commonly used call types are exported as constants in this package and should be used whenever possible. The most common call types are http (trace.CallTypeHTTP) and grpc (trace.CallTypeGRPC).

Use the extra args to add stuff to traces and logs and these can have more information as needed (including actual queries for instance).

The log includes a initial Debug entry and a final Error entry if the call failed (but no IDs entry if the call succeeded). Success or failure is determined by whether there was a SetCallStatus or not. (Panics detected in EndCall are considered errors).

StartCalls can be nested.

func StartSpan

func StartSpan(ctx context.Context, name string, args ...log.Marshaler) context.Context

StartSpan starts a new span.

Use trace.End to end this.

func StartSpanAsync

func StartSpanAsync(ctx context.Context, name string, args ...log.Marshaler) context.Context

Deprecated: use StartSpan() instead. It will handle async traces automatically. StartSpanAsync starts a new async span.

An async span does not have to complete before the parent span completes.

Use trace.End to end this.

func StartTrace

func StartTrace(ctx context.Context, name string) context.Context

Deprecated: use StartSpan() instead. It will start a trace automatically if the context does not contain one already. StartTrace starts a new root span/trace.

Use trace.End to end this.

func StartTracing

func StartTracing(serviceName string) error

Deprecated: Use InitTracer() instead. StartTracing starts the tracing infrastructure.

This should be called at the start of the application.

func ToHeaders

func ToHeaders(ctx context.Context) map[string][]string

ToHeaders writes the current trace context into a headers map

Only use for GRPC. Prefer NewTransport for http calls.

func WithScheduledTime

func WithScheduledTime(t time.Time) call.Option

WithScheduledTime set the call Info scheduled at time

Types

type Annotator

type Annotator struct {
	// contains filtered or unexported fields
}

Annotator is a SpanProcessor that adds service-level tags on every span

func (Annotator) ForceFlush

func (a Annotator) ForceFlush(context.Context) error

func (Annotator) OnEnd

func (a Annotator) OnEnd(s sdktrace.ReadOnlySpan)

func (Annotator) OnStart

func (a Annotator) OnStart(_ context.Context, s sdktrace.ReadWriteSpan)

func (Annotator) Shutdown

func (a Annotator) Shutdown(context.Context) error

type Config

type Config struct {
	Otel       `yaml:"OpenTelemetry"`
	GlobalTags `yaml:"GlobalTags,omitempty"`
}

tracing config goes into trace.yaml

func (*Config) Load

func (c *Config) Load() error

type GlobalTags

type GlobalTags struct {
	DevEmail string `yaml:"DevEmail,omitempty"`
}

func (*GlobalTags) MarshalLog

func (g *GlobalTags) MarshalLog(addField func(key string, v interface{}))

type Handler

type Handler struct {
	// contains filtered or unexported fields
}

func (*Handler) ServeHTTP

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type Otel

type Otel struct {
	// Enabled determines whether to turn on tracing
	Enabled bool `yaml:"Enabled"`
	// Endpoint for the tracing backend
	Endpoint string `yaml:"Endpoint"`
	// AdditionalEndpoint another backend for tracing
	AdditionalEndpoint string `yaml:"AdditionalEndpoint"`
	// Dataset the honeycomb grouping of traces
	Dataset string `yaml:"Dataset"`
	// SamplePercent the rate at which to sample
	SamplePercent float64 `yaml:"SamplePercent"`
	// Debug allows printing debug statements for traces
	Debug bool `yaml:"Debug"`
	// Stdout also outputs traces to stdout
	Stdout bool `yaml:"Stdout"`
	// APIKey used for authentication with the backend at Endpoint
	APIKey cfg.Secret `yaml:"APIKey"`
	// AdditionalAPIKey used for authentication with the backend at AdditionalEndpoint
	AdditionalAPIKey cfg.Secret `yaml:"AdditionalAPIKey"`
}