Documentation
¶
Overview ¶
Package tracing is the primary entrypoint into LabKit's distributed tracing functionality.
(This documentation assumes some minimal knowledge of Distributed Tracing, and uses tracing terminology without providing definitions. Please review https://opentracing.io/docs/overview/what-is-tracing/ for an broad overview of distributed tracing if you are not familiar with the technology)
Internally the `tracing` package relies on Opentracing, but avoids leaking this abstraction. In theory, LabKit could replace Opentracing with another distributed tracing interface, such as Zipkin or OpenCensus, without needing to make changes to the application (other than updating the LabKit dependency, of course).
This design decision is deliberate: the package should not leak the underlying tracing implementation.
The package provides three primary exports:
* `tracing.Initialize()` for initializing the global tracer using the `GITLAB_TRACING` environment variable. * An HTTP Handler middleware, `tracing.Handler()`, for instrumenting incoming HTTP requests. * An HTTP RoundTripper, `tracing.NewRoundTripper()` for instrumenting outbound HTTP requests to other services.
The provided example in `example_test.go` demonstrates usage of both the HTTP Middleware and the HTTP RoundTripper.
*Initializing the global tracer*
Opentracing makes use of a global tracer. Opentracing ships with a default NoOp tracer which does nothing at all. This is always configured, meaning that, without initialization, Opentracing does nothing and has a very low overhead.
LabKit's tracing is configured through an environment variable, `GITLAB_TRACING`. This environment variable contains a "connection string"-like configuration, such as:
* `opentracing://jaeger?udp_endpoint=localhost:6831` * `opentracing://datadog` * `opentracing://lightstep` * `opentracing://stackdriver?sampler_probability=0.001&project_id=gitlab-pre`
The parameters for these connection-strings are implementation specific.
This configuration is identical to the one used to configure GitLab's ruby tracing libraries in the `Gitlab::Tracing` package. Having a consistent configuration makes it easy to configure multiple processes at the same time. For example, in GitLab Development Kit, tracing can be configured with a single environment variable, `GITLAB_TRACING=... gdk run`, since `GITLAB_TRACING` will configure Workhorse (written in Go), Gitaly (written in Go) and GitLab's rails components, using the same configuration.
*Compiling applications with Tracing support*
Go's Opentracing interface does not allow tracing implementations to be loaded dynamically; implementations need to be compiled into the application. With LabKit, this is done conditionally, using build tags. Two build tags need to be specified:
* `tracer_static` - this compiles in the static plugin registry support * `tracer_static_[DRIVER_NAME]` - this compile in support for the given driver.
For example, to compile support for Jaeger, compile your Go app with `tracer_static,tracer_static_jaeger`
Note that multiple (or all) drivers can be compiled in alongside one another: using the tags: `tracer_static,tracer_static_jaeger,tracer_static_lightstep,tracer_static_datadog,tracer_static_stackdriver`
If the `GITLAB_TRACING` environment variable references an unknown or unregistered driver, it will log a message and continue without tracing. This is a deliberate decision: the risk of bringing down a cluster during a rollout with a misconfigured tracer configuration is greater than the risk of an operator loosing some time because their application was not compiled with the correct tracers.
*Using the HTTP Handler middleware to instrument incoming HTTP requests*
When an incoming HTTP request arrives on the server, it may already include Distributed Tracing headers, propagated from an upstream service.
The tracing middleware will attempt to extract the tracing information from the headers (the exact headers used are tracing implementation specific), set up a span and pass the information through the request context.
It is up to the Opentracing implementation to decide whether the span will be sent to the tracing infrastructure. This will be implementation-specific, but generally relies on server load, sampler configuration, whether an error occurred, whether certain spans took an anomalous amount of time, etc.
*Using the HTTP RoundTripper to instrument outgoing HTTP requests*
The RoundTripper should be added to the HTTP client RoundTripper stack (see the example). When an outbound HTTP request is sent from the HTTP client, the RoundTripper will determine whether there is an active span and if so, will inject headers into the outgoing HTTP request identifying the span. The details of these headers is implementation specific.
It is important to ensure that the context is passed into the outgoing request, using `req.WithContext(ctx)` so that the correct span information can be injected into the request headers.
*Propagating tracing information to child processes*
Sometimes we want a trace to continue from a parent process to a spawned child process. For this, the tracing package provides `tracing.NewEnvInjector()` and `tracing.ExtractFromEnv()`, for the parent and child processes respectively.
NewEnvInjector() will configure a []string array of environment variables, ensuring they have the correct tracing configuration and any trace and span identifiers. NewEnvInjector() should be called in the child process and will extract the trace and span information from the environment.
Please review the examples in the godocs for details of how to implement both approaches.
Example ¶
This example shows how to initialize tracing and then wrap all incoming calls.
package main
import (
"fmt"
"io"
"net/http"
"time"
log "github.com/sirupsen/logrus"
"gitlab.com/gitlab-org/labkit/tracing"
)
func main() {
// Tell the tracer to initialize as service "gitlab-wombat"
tracing.Initialize(tracing.WithServiceName("gitlab-wombat"))
tr := &http.Transport{
MaxIdleConns: 10,
IdleConnTimeout: 30 * time.Second,
DisableCompression: true,
}
client := &http.Client{
Transport: tracing.NewRoundTripper(tr),
}
// Listen and propagate traces
http.Handle("/foo",
// Add the tracing middleware in
tracing.Handler(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
req, err := http.NewRequest(http.MethodGet, "http://localhost:8080/bar", nil)
if err != nil {
w.WriteHeader(500)
return
}
req = req.WithContext(r.Context())
resp, err := client.Do(req)
if err != nil {
w.WriteHeader(500)
return
}
defer resp.Body.Close()
_, err = io.Copy(w, resp.Body)
if err != nil {
w.WriteHeader(500)
return
}
}),
// Use this route identifier with the tracing middleware
tracing.WithRouteIdentifier("/foo"),
))
// Listen and propagate traces
http.Handle("/bar",
// Add the tracing middleware in
tracing.Handler(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "bar")
}),
// Use this route identifier with the tracing middleware
tracing.WithRouteIdentifier("/bar"),
))
log.Fatal(http.ListenAndServe(":0", nil))
}
Output:
Index ¶
- Variables
- func ExtractFromEnv(ctx context.Context, opts ...ExtractFromEnvOption) (context.Context, func())
- func Handler(h http.Handler, opts ...HandlerOption) http.Handler
- func Initialize(opts ...InitializationOption) io.Closer
- func IsSampled(span opentracing.Span) bool
- func NewRoundTripper(delegate http.RoundTripper, opts ...RoundTripperOption) http.RoundTripper
- type EnvInjector
- type EnvInjectorOption
- type ExtractFromEnvOption
- type HandlerOption
- type InitializationOption
- type OperationNamer
- type RoundTripperOption
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrConfiguration = fmt.Errorf("tracing is not properly configured")
ErrConfiguration is returned when the tracer is not properly configured.
Functions ¶
func ExtractFromEnv ¶
func ExtractFromEnv(ctx context.Context, opts ...ExtractFromEnvOption) (context.Context, func())
ExtractFromEnv will extract a span from the environment after it has been passed in from the parent process. Returns a new context, and a defer'able function, which should be called on process termination.
Example ¶
package main
import (
"context"
"gitlab.com/gitlab-org/labkit/tracing"
)
func main() {
// Tell the tracer to initialize as service "gitlab-child-process"
tracing.Initialize(tracing.WithServiceName("gitlab-child-process"))
ctx, finished := tracing.ExtractFromEnv(context.Background())
defer finished()
// Program execution happens here...
func(_ context.Context) {}(ctx)
}
Output:
func Handler ¶
func Handler(h http.Handler, opts ...HandlerOption) http.Handler
Handler will extract tracing from inbound request.
func Initialize ¶
func Initialize(opts ...InitializationOption) io.Closer
Initialize will initialize distributed tracing.
func IsSampled ¶ added in v1.18.0
func IsSampled(span opentracing.Span) bool
IsSampled returns the sampling status (true/false) of a span. This function wraps around the actual implementation in `impl` packet. Technically, we don't need this wrapper, but the `impl` package contains detailed implementation. Most consumers import `gitlab.com/gitlab-org/labkit/tracing`
func NewRoundTripper ¶
func NewRoundTripper(delegate http.RoundTripper, opts ...RoundTripperOption) http.RoundTripper
NewRoundTripper acts as a "client-middleware" for outbound http requests adding instrumentation to the outbound request and then delegating to the underlying transport.
Types ¶
type EnvInjector ¶
EnvInjector will inject tracing information into an environment in preparation for spawning a child process. This includes trace and span identifiers, as well as the GITLAB_TRACING configuration. Will gracefully degrade if tracing is not configured, or an active span is not currently available.
func NewEnvInjector ¶
func NewEnvInjector(opts ...EnvInjectorOption) EnvInjector
NewEnvInjector will create a new environment injector.
Example ¶
package main
import (
"context"
"os/exec"
log "github.com/sirupsen/logrus"
"gitlab.com/gitlab-org/labkit/tracing"
)
func main() {
envInjector := tracing.NewEnvInjector()
cmd := exec.Command("ls")
env := []string{
"FOO=bar",
}
// envInjector will inject any required values
cmd.Env = envInjector(context.Background(), env)
if err := cmd.Run(); err != nil {
log.WithError(err).Fatal("Command failed")
}
}
Output:
type EnvInjectorOption ¶
type EnvInjectorOption func(*envInjectorConfig)
EnvInjectorOption will configure an environment injector.
type ExtractFromEnvOption ¶
type ExtractFromEnvOption func(*extractFromEnvConfig)
ExtractFromEnvOption will configure an environment injector.
type HandlerOption ¶
type HandlerOption func(*handlerConfig)
HandlerOption will configure a correlation handler.
func WithRouteIdentifier ¶
func WithRouteIdentifier(routeIdentifier string) HandlerOption
WithRouteIdentifier allows a RouteIdentifier attribute to be set in the handler. This value will appear in the traces.
type InitializationOption ¶
type InitializationOption func(*initializationConfig)
InitializationOption will configure a correlation handler.
func WithConnectionString ¶
func WithConnectionString(connectionString string) InitializationOption
WithConnectionString allows the opentracing connection string to be overridden. By default this will be retrieved from the GITLAB_TRACING environment variable.
func WithServiceName ¶
func WithServiceName(serviceName string) InitializationOption
WithServiceName allows the service name to be configured for the tracer this will appear in traces.
type OperationNamer ¶
OperationNamer will return an operation name given an HTTP request.
type RoundTripperOption ¶
type RoundTripperOption func(*roundTripperConfig)
RoundTripperOption will configure a correlation handler.
Source Files
Directories