trace

Documentation

Overview

package trace provides activity tracing via ctx through Tasks and Spans

Basic Concepts

Tracing can be used to identify where a piece of code spends its time.

The Go standard library provides package runtime/trace which is useful to identify CPU bottlenecks or to understand what happens inside the Go runtime. However, it is not ideal for application level tracing, in particular if those traces should be understandable to tech-savvy users (albeit not developers).

This package provides the concept of Tasks and Spans to express what activity is happening within an application:

• Neither task nor span is really tangible but instead contained within the context.Context tree
• Tasks represent concurrent activity (i.e. goroutines).
• Spans represent a semantic stack trace within a task.

As a consequence, whenever a context is propagated across goroutine boundary, you need to create a child task:

go func(ctx context.Context) {
  ctx, endTask = WithTask(ctx, "what-happens-inside-the-child-task")
  defer endTask()
  // ...
}(ctx)

Within the task, you can open up a hierarchy of spans. In contrast to tasks, which have can multiple concurrently running child tasks, spans must nest and not cross the goroutine boundary.

ctx, endSpan = WithSpan(ctx, "copy-dir")
defer endSpan()
for _, f := range dir.Files() {
  func() {
    ctx, endSpan := WithSpan(ctx, fmt.Sprintf("copy-file %q", f))
    defer endspan()
    b, _ := ioutil.ReadFile(f)
    _ = ioutil.WriteFile(f + ".copy", b, 0600)
  }()
}

In combination:

ctx, endTask = WithTask(ctx, "copy-dirs")
defer endTask()
for i := range dirs {
  go func(dir string) {
    ctx, endTask := WithTask(ctx, "copy-dir")
    defer endTask()
    for _, f := range filesIn(dir) {
      func() {
        ctx, endSpan := WithSpan(ctx, fmt.Sprintf("copy-file %q", f))
        defer endspan()
        b, _ := ioutil.ReadFile(f)
        _ = ioutil.WriteFile(f + ".copy", b, 0600)
      }()
    }
  }()
}

Note that a span ends at the time you call endSpan - not before and not after that. If you violate the stack-like nesting of spans by forgetting an endSpan() invocation, the out-of-order endSpan() will panic.

A similar rule applies to the endTask closure returned by WithTask: If a task has live child tasks at the time you call endTask(), the call will panic.

Recovering from endSpan() or endTask() panics will corrupt the trace stack and lead to corrupt tracefile output.

Best Practices For Naming Tasks And Spans

Tasks should always have string constants as names, and must not contain the `#` character. WHy? First, the visualization by chrome://tracing draws a horizontal bar for each task in the trace. Also, the package appends `#NUM` for each concurrently running instance of a task name. Note that the `#NUM` suffix will be reused if a task has ended, in order to avoid an infinite number of horizontal bars in the visualization.

Chrome-compatible Tracefile Support

The activity trace generated by usage of WithTask and WithSpan can be rendered to a JSON output file that can be loaded into chrome://tracing . Apart from function GetSpanStackOrDefault, this is the main benefit of this package.

First, there is a convenience environment variable 'ZREPL_ACTIVITY_TRACE' that can be set to an output path. From process start onward, a trace is written to that path.

More consumers can attach to the activity trace through the ChrometraceClientWebsocketHandler websocket handler.

If a write error is encountered with any consumer (including the env-var based one), the consumer is closed and will not receive further trace output.

Index

• Variables
• func ChrometraceClientWebsocketHandler(conn *websocket.Conn)
• func GetSpanStackOrDefault(ctx context.Context, kind StackKind, def string) string
• func RegisterCallback(c Callback)
• func RegisterMetrics(r prometheus.Registerer)
• func WithInherit(ctx, inheritFrom context.Context) context.Context
• type Callback
• type DoneFunc
  • func WithSpan(ctx context.Context, annotation string) (context.Context, DoneFunc)
  • func WithSpanFromStackUpdateCtx(ctx *context.Context) DoneFunc
  • func WithTask(ctx context.Context, taskName string) (context.Context, DoneFunc)
  • func WithTaskAndSpan(ctx context.Context, task string, span string) (context.Context, DoneFunc)
  • func WithTaskFromStack(ctx context.Context) (context.Context, DoneFunc)
  • func WithTaskFromStackUpdateCtx(ctx *context.Context) DoneFunc
  • func WithTaskGroup(ctx context.Context, taskGroup string) (_ context.Context, add func(f func(context.Context)), waitEnd DoneFunc)
• type SpanInfo
• type StackKind

Variables

var (
	StackKindId = &StackKind{
		symbolizeTask: func(t *traceNode) string { return t.id },
		symbolizeSpan: func(s *traceNode) string { return s.id },
	}
	SpanStackKindCombined = &StackKind{
		symbolizeTask: func(t *traceNode) string { return fmt.Sprintf("(%s %q)", t.id, t.annotation) },
		symbolizeSpan: func(s *traceNode) string { return fmt.Sprintf("(%s %q)", s.id, s.annotation) },
	}
	SpanStackKindAnnotation = &StackKind{
		symbolizeTask: func(t *traceNode) string { return t.annotation },
		symbolizeSpan: func(s *traceNode) string { return s.annotation },
	}
)

var ErrAlreadyActiveChildSpan = fmt.Errorf("create child span: span already has an active child span")

var ErrSpanStillHasActiveChildSpan = fmt.Errorf("end span: span still has active child spans")

var ErrTaskStillHasActiveChildTasks = fmt.Errorf("end task: task still has active child tasks")

Functions

func ChrometraceClientWebsocketHandler

func ChrometraceClientWebsocketHandler(conn *websocket.Conn)

func GetSpanStackOrDefault

func GetSpanStackOrDefault(ctx context.Context, kind StackKind, def string) string

func RegisterCallback

func RegisterCallback(c Callback)

func RegisterMetrics

func RegisterMetrics(r prometheus.Registerer)

func WithInherit

func WithInherit(ctx, inheritFrom context.Context) context.Context

WithInherit inherits the task hierarchy from inheritFrom into ctx. The returned context is a child of ctx, but its task and span are those of inheritFrom.

Note that in most use cases, callers most likely want to call WithTask since it will most likely be in some sort of connection handler context.

Types

type Callback

type Callback struct {
	OnBegin func(ctx context.Context)
	OnEnd   func(ctx context.Context, spanInfo SpanInfo)
}

type DoneFunc

type DoneFunc func()

Returned from WithTask or WithSpan. Must be called once the task or span ends. See package-level docs for nesting rules. Wrong call order / forgetting to call it will result in panics.

func WithSpan

func WithSpan(ctx context.Context, annotation string) (context.Context, DoneFunc)

Start a new span. Important: ctx must have an active task (see WithTask)

func WithSpanFromStackUpdateCtx

func WithSpanFromStackUpdateCtx(ctx *context.Context) DoneFunc

use like this:

defer WithSpanFromStackUpdateCtx(&existingCtx)()

func WithTask

func WithTask(ctx context.Context, taskName string) (context.Context, DoneFunc)

Start a new root task or create a child task of an existing task.

This is required when starting a new goroutine and passing an existing task context to it.

taskName should be a constantand must not contain '#'

The implementation ensures that, if multiple tasks with the same name exist simultaneously, a unique suffix is appended to uniquely identify the task opened with this function.

func WithTaskAndSpan

func WithTaskAndSpan(ctx context.Context, task string, span string) (context.Context, DoneFunc)

create a task and a span within it in one call

func WithTaskFromStack

func WithTaskFromStack(ctx context.Context) (context.Context, DoneFunc)

derive task name from call stack (caller's name)

func WithTaskFromStackUpdateCtx

func WithTaskFromStackUpdateCtx(ctx *context.Context) DoneFunc

derive task name from call stack (caller's name) and update *ctx to point to be the child task ctx

func WithTaskGroup

func WithTaskGroup(ctx context.Context, taskGroup string) (_ context.Context, add func(f func(context.Context)), waitEnd DoneFunc)

create a span during which several child tasks are spawned using the `add` function

IMPORTANT FOR USERS: Caller must ensure that the capturing behavior is correct, the Go linter doesn't catch this.

type SpanInfo

type SpanInfo interface {
	StartedAt() time.Time
	EndedAt() time.Time
	TaskAndSpanStack(kind *StackKind) string
}

type StackKind

type StackKind struct {
	// contains filtered or unexported fields
}