otelpyroscope

README

Profiling Instrumentation for OpenTelemetry Go SDK

NOTE: This is an experimental package -- and will be officially supported in future versions of Pyroscope

The package provides means to integrate tracing with profiling. More specifically, a TracerProvider implementation, that annotates profiling data with span IDs: when a new trace span emerges, the tracer adds a span_id pprof tag that points to the span. This makes it possible to filter out a profile of a particular trace span in Pyroscope.

Note that the module does not control pprof profiler itself – it still needs to be started for profiles to be collected. This can be done either via runtime/pprof package, or using the Pyroscope client.

By default, only the root span gets labeled (the first span created locally): such spans are marked with the pyroscope.profile.id attribute set to the span ID. Please note that presence of the attribute does not necessarily indicate that the span has a profile: stack trace samples might not be collected, if the utilized CPU time is less than the sample interval (10ms).

Limitations:

• Only CPU profiling is fully supported at the moment.

Trace spans profiles

To start profiling trace spans, you need to include our go module in your app:

go get github.com/grafana/otel-profiling-go

Then add the pyroscope tracer provider:

package main

import (
	otelpyroscope "github.com/grafana/otel-profiling-go"
	"github.com/grafana/pyroscope-go"
)

func main() {
	// Initialize your tracer provider as usual.
	tp := initTracer()

	// Wrap it with otelpyroscope tracer provider.
	otel.SetTracerProvider(otelpyroscope.NewTracerProvider(tp))

	// If you're using Pyroscope Go SDK, initialize pyroscope profiler.
	_, _ = pyroscope.Start(pyroscope.Config{
		ApplicationName: "my-service",
		ServerAddress:   "http://localhost:4040",
	})

	// Your code goes here.
}

Tracing integration is supported in pull mode as well: if you scrape profiles using Grafana Agent, you should make sure that the pyroscope service_name label matches service.name attribute specified in the OTel SDK configuration. Please refer to the Grafana Agent documentation to learn more.

Example

You can find a complete example setup with Grafana Tempo in the Pyroscope repository.

Documentation

Index

• func NewTracerProvider(tp trace.TracerProvider, options ...Option) trace.TracerProvider
• type Config
• type Option
  • func WithAddSpanName(bool) Option
  • func WithAppName(string) Option
  • func WithDefaultProfileURLBuilder(_, _ string) Option
  • func WithProfileBaselineLabels(map[string]string) Option
  • func WithProfileBaselineURL(bool) Option
  • func WithProfileURL(bool) Option
  • func WithProfileURLBuilder(func(_ string) string) Option
  • func WithPyroscopeURL(string) Option
  • func WithRootSpanOnly(bool) Option

Functions

func NewTracerProvider

func NewTracerProvider(tp trace.TracerProvider, options ...Option) trace.TracerProvider

NewTracerProvider creates a new tracer provider that annotates pprof samples with span_id label. This allows to establish a relationship between pprof profiles and reported tracing spans.

Types

type Config

type Config struct {
	AppName                   string
	PyroscopeURL              string
	IncludeProfileURL         bool
	IncludeProfileBaselineURL bool
	ProfileBaselineLabels     map[string]string

	RootOnly    bool
	AddSpanName bool
}

Config describes tracer configuration. DEPRECATED: Do not use.

type Option

type Option func(*tracerProvider)

func WithAddSpanName

func WithAddSpanName(bool) Option

WithAddSpanName specifies whether the current span name should be added to the profile labels. N.B if the name is dynamic, or too many values are supposed, this may significantly deteriorate performance. By default, span name is not added to profile labels. DEPRECATED: Ignored by tracer.

func WithAppName

func WithAppName(string) Option

WithAppName specifies the profiled application name. It should match the name specified in pyroscope configuration. Required, if profile URL or profile baseline URL is enabled. DEPRECATED: Ignored by tracer.

func WithDefaultProfileURLBuilder

func WithDefaultProfileURLBuilder(_, _ string) Option

WithDefaultProfileURLBuilder specifies the default profile URL builder. DEPRECATED: Ignored by tracer.

func WithProfileBaselineLabels

func WithProfileBaselineLabels(map[string]string) Option

WithProfileBaselineLabels provides a map of extra labels to be added to the baseline query alongside with pprof labels set in runtime. Typically, it should match the labels specified in the Pyroscope profiler config. Note that the map must not be modified. DEPRECATED: Ignored by tracer.

func WithProfileBaselineURL

func WithProfileBaselineURL(bool) Option

WithProfileBaselineURL specifies whether to add the pyroscope.profile.baseline.url attribute with the URL to the baseline profile. See WithProfileBaselineLabels. DEPRECATED: Ignored by tracer.

func WithProfileURL

func WithProfileURL(bool) Option

WithProfileURL specifies whether to add the pyroscope.profile.url attribute with the URL to the span profile. DEPRECATED: Ignored by tracer.

func WithProfileURLBuilder

func WithProfileURLBuilder(func(_ string) string) Option

WithProfileURLBuilder specifies how profile URL is to be built. DEPRECATED: Ignored by tracer.

func WithPyroscopeURL

func WithPyroscopeURL(string) Option

WithPyroscopeURL provides a base URL for the profile and baseline URLs. Required, if profile URL or profile baseline URL is enabled. DEPRECATED: Ignored by tracer.

func WithRootSpanOnly

func WithRootSpanOnly(bool) Option

WithRootSpanOnly indicates that only the root span is to be profiled. The profile includes samples captured during child span execution but the spans won't have their own profiles and won't be annotated with pyroscope.profile attributes. The option is enabled by default. DEPRECATED: Ignored by tracer.