grpc

README

gRPC Remote Storage Plugins

Update (May 2024): as of v1.58, Jaeger will no longer support grpc-sidecar plugin model. Only gRPC Remote Storage API is suppored.

In order to support an ecosystem of different backend storage implementations, Jaeger supports a gRPC-based Remote Strorage API. The custom backend storage solutions need to implement storage_v1 gRPC interfaces defined in plugin/storage/grpc/proto/.

+----------------------------------+                  +-----------------------------+
|                                  |                  |                             |
|                  +-------------+ |       gRPC       | +-------------+             |
|                  |             | |                  | |             |             |
| jaeger-component | grpc-client +----------------------> grpc-server | custom impl |
|                  |             | |                  | |             |             |
|                  +-------------+ |                  | +-------------+             |
|                                  |                  |                             |
+----------------------------------+                  +-----------------------------+

    Jaeger official components                         Custom backend implementation

Implementing a plugin

Although the instructions below are limited to Go, plugins can be implemented any language. Languages other than Go would implement a gRPC server using the storage_v1 proto interfaces. The proto file can be found in plugin/storage/grpc/proto/. To generate the bindings for your language you would use protoc with the appropriate xx_out= flag. This is detailed in the protobuf documentation and you can see an example of how it is done for Go in the top level Jaeger Makefile.

The easiest way to generate the gRPC storage plugin bindings is to use Docker Protobuf which is a lightweight protoc Docker image containing the dependencies needed to generate code for multiple languages. For example, one can generate bindings for C# on Windows with Docker for Windows using the following steps:

1. First clone the Jaeger github repo to a folder (e.g. c:\source\repos\jaeger):

$ mkdir c:\source\repos\jaeger
$ cd c:\source\repos\jaeger
$ git clone https://github.com/jaegertracing/jaeger.git c:\source\repos\jaeger

1. Initialize the Jaeger repo submodules (this pulls in code from other Jaeger repositories that are needed):

$ git submodule update --init --recursive

1. Then execute the following Docker command which mounts the local directory c:\source\repos\jaeger to the directory /jaeger in the Docker container and then executes the jaegertracing/protobuf:0.2.0 command. This will create a file called Storage.cs in your local Windows folder c:\source\repos\jaeger\code containing the gRPC Storage API bindings.

$ docker run --rm -u 1000 -v/c/source/repos/jaeger:/jaeger -w/jaeger \
    jaegertracing/protobuf:0.2.0 "-I/jaeger -Iidl/proto/api_v2 -I/usr/include/github.com/gogo/protobuf -Iplugin/storage/grpc/proto --csharp_out=/jaeger/code plugin/storage/grpc/proto/storage.proto"

An example of a Go binary that implements Remote Storage API can be found in cmd/remote-storage. That specific binary does not implement a custom backend, instead it supports the same backend implementations as available directly in Jaeger, but it makes them accessible via a Remote Storage API (and is being used in the integration tests).

The API consists of several gRPC services:

• SpanReaderPlugin - used for querying the data
• SpanWriterPlugin - used for writing data
• (optional) StreamingSpanWriterPlugin - allows more efficient transmission
• (optional) ArchiveSpanWriterPlugin and ArchiveSpanReaderPlugin - to support archiving storage
• (optional) DependenciesReaderPlugin - for reading service dependencies
• (optional) PluginCapabilities - can be interrogated to find out which services an implementation supports

The API supports writing spans via gRPC stream, instead of unary messages. Streaming writes can improve throughput and decrease CPU load (see benchmarks in Issue #3636). The backend needs to implement StreamingSpanWriterPlugin service and indicate support via the streamingSpanWriter flag in the Capabilities response.

Note that using the streaming spanWriter may make the collector's save_by_svr metric inaccurate, in which case users will need to pay attention to the metrics provided by the plugin.

Certifying compliance

A plugin implementation shall verify it's correctness with Jaeger storage protocol by running the storage integration tests from integration package.

import (
	jaeger_integration_tests "github.com/jaegertracing/jaeger/plugin/storage/integration"
)

func TestJaegerStorageIntegration(t *testing.T) {
        ...
	si := jaeger_integration_tests.StorageIntegration{
		SpanReader: createSpanReader(),
		SpanWriter: createSpanWriter(),
		CleanUp: func() error { ... },
		Refresh: func() error { ... },
		SkipList: []string {  // Skip any unsupported tests
		},
	}
	// Runs all storage integration tests.
	si.IntegrationTestAll(t)
}

For more details, refer to one of the following implementations.

1. jaeger-clickhouse
2. Timescale DB via Promscale

Tracing

Jaeger requests to the backend implementation will include standard OTEL tracing headers. The implementation may choose to participate in those traces to allow end to end visibility of Jaeger's own operations (typically only enabled for read requests, as tracing write requests results in traces for traces and may cause infinite loops).

Bearer token propagation from the UI

When using --query.bearer-token-propagation=true, the bearer token will be properly passed on to the gRPC plugin server. To get access to the bearer token in your plugin, use a method similar to:

import (
    // ... other imports
    "fmt"
    "google.golang.org/grpc/metadata"

    "github.com/jaegertracing/jaeger/plugin/storage/grpc"
)

// ... spanReader type declared here

func (r *spanReader) extractBearerToken(ctx context.Context) (string, bool) {
	if md, ok := metadata.FromIncomingContext(ctx); ok {
		values := md.Get(grpc.BearerTokenKey)
		if len(values) > 0 {
			return values[0], true
		}
	}
	return "", false
}

// ... spanReader interface implementation

func (r *spanReader) GetServices(ctx context.Context) ([]string, error) {
    str, ok := r.extractBearerToken(ctx)
    fmt.Println(fmt.Sprintf("spanReader.GetServices: bearer-token: '%s', wasGiven: '%t'" str, ok))
    // ...
}

Documentation

Index

• type ClientPluginServices
  • func (c *ClientPluginServices) Close() error
• type ConfigV2
  • func (c *ConfigV2) Build(logger *zap.Logger, tracerProvider trace.TracerProvider) (*ClientPluginServices, error)
• type Configuration
  • func (c *Configuration) TranslateToConfigV2() *ConfigV2
• type Factory
  • func NewFactory() *Factory
  • func NewFactoryWithConfig(cfg ConfigV2, metricsFactory metrics.Factory, logger *zap.Logger) (*Factory, error)
  • func (*Factory) AddFlags(flagSet *flag.FlagSet)
  • func (f *Factory) Close() error
  • func (f *Factory) CreateArchiveSpanReader() (spanstore.Reader, error)
  • func (f *Factory) CreateArchiveSpanWriter() (spanstore.Writer, error)
  • func (f *Factory) CreateDependencyReader() (dependencystore.Reader, error)
  • func (f *Factory) CreateSpanReader() (spanstore.Reader, error)
  • func (f *Factory) CreateSpanWriter() (spanstore.Writer, error)
  • func (f *Factory) InitFromViper(v *viper.Viper, logger *zap.Logger)
  • func (f *Factory) Initialize(metricsFactory metrics.Factory, logger *zap.Logger) error

Types

type ClientPluginServices

type ClientPluginServices struct {
	shared.PluginServices
	Capabilities shared.PluginCapabilities
	// contains filtered or unexported fields
}

ClientPluginServices defines services plugin can expose and its capabilities

func (*ClientPluginServices) Close

func (c *ClientPluginServices) Close() error

type ConfigV2

type ConfigV2 struct {
	Tenancy                        tenancy.Options `mapstructure:"multi_tenancy"`
	configgrpc.ClientConfig        `mapstructure:",squash"`
	exporterhelper.TimeoutSettings `mapstructure:",squash"`
}

func (*ConfigV2) Build

func (c *ConfigV2) Build(logger *zap.Logger, tracerProvider trace.TracerProvider) (*ClientPluginServices, error)

TODO move this to factory.go

type Configuration

type Configuration struct {
	RemoteServerAddr     string `yaml:"server" mapstructure:"server"`
	RemoteTLS            tlscfg.Options
	RemoteConnectTimeout time.Duration `yaml:"connection-timeout" mapstructure:"connection-timeout"`
	TenancyOpts          tenancy.Options
}

Configuration describes the options to customize the storage behavior.

func (*Configuration) TranslateToConfigV2

func (c *Configuration) TranslateToConfigV2() *ConfigV2

type Factory

type Factory struct {
	// contains filtered or unexported fields
}

Factory implements storage.Factory and creates storage components backed by a storage plugin.

func NewFactory

func NewFactory() *Factory

NewFactory creates a new Factory.

func NewFactoryWithConfig

func NewFactoryWithConfig(
	cfg ConfigV2,
	metricsFactory metrics.Factory,
	logger *zap.Logger,
) (*Factory, error)

NewFactoryWithConfig is used from jaeger(v2).

func (*Factory) AddFlags

func (*Factory) AddFlags(flagSet *flag.FlagSet)

AddFlags implements plugin.Configurable

func (*Factory) Close

func (f *Factory) Close() error

Close closes the resources held by the factory

func (*Factory) CreateArchiveSpanReader

func (f *Factory) CreateArchiveSpanReader() (spanstore.Reader, error)

CreateArchiveSpanReader implements storage.ArchiveFactory

func (*Factory) CreateArchiveSpanWriter

func (f *Factory) CreateArchiveSpanWriter() (spanstore.Writer, error)

CreateArchiveSpanWriter implements storage.ArchiveFactory

func (*Factory) CreateDependencyReader

func (f *Factory) CreateDependencyReader() (dependencystore.Reader, error)

CreateDependencyReader implements storage.Factory

func (*Factory) CreateSpanReader

func (f *Factory) CreateSpanReader() (spanstore.Reader, error)

CreateSpanReader implements storage.Factory

func (*Factory) CreateSpanWriter

func (f *Factory) CreateSpanWriter() (spanstore.Writer, error)

CreateSpanWriter implements storage.Factory

func (*Factory) InitFromViper

func (f *Factory) InitFromViper(v *viper.Viper, logger *zap.Logger)

InitFromViper implements plugin.Configurable

func (*Factory) Initialize

func (f *Factory) Initialize(metricsFactory metrics.Factory, logger *zap.Logger) error

Initialize implements storage.Factory