otelgo

README

Integrating OpenTelemetry (OTEL) with Your Application

Overview

OpenTelemetry (OTEL) provides a set of tools, APIs, and SDKs to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for monitoring and troubleshooting your applications. This guide will help you integrate OTEL into your application, enabling you to capture and export metrics and traces.

Setting up the OTEL Collector Service on GCP

This guide provides step-by-step instructions to set up the OpenTelemetry (OTEL) Collector Service on Google Cloud Platform (GCP) using the provided configuration file.

Prerequisites

Before enabling OTEL in your application, ensure that the collector service is up and running. The collector service is responsible for capturing and exporting the telemetry data. By default, OTEL is disabled in the configuration.

1. GCP Account: Ensure you have a GCP account and have access to a GCP project.
2. Google Cloud SDK: Install the Google Cloud SDK.
3. Docker: Install Docker to run the OTEL Collector in a container.
4. Kubernetes Cluster (GKE): Set up a Google Kubernetes Engine (GKE) cluster if deploying in a Kubernetes environment.

Steps to Set Up and Configure the OTEL Collector Service with Proxy Adaptor.

Step 1: Use the below config.yaml (or set up the CONFIG_FILE environment variable) file for OTEL collector service

  # Enable the health check in the proxy application config only if the
  # "health_check" extension is added to this OTEL config for the collector service.
  #
  # Recommendation:
  # Enable the OTEL health check if you need to verify the collector's availability
  # at the start of the application. For development or testing environments, it can
  # be safely disabled to reduce complexity.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:55681"

processors:
  batch:
    send_batch_max_size: 0
    send_batch_size: 8192
    timeout: 5s

  memory_limiter:
    # drop metrics if memory usage gets too high
    check_interval: 5s
    limit_percentage: 65
    spike_limit_percentage: 20

  resourcedetection:
    detectors: [gcp]
    timeout: 10s

exporters:
  googlecloud:
    metric:
      instrumentation_library_labels: true
      service_resource_labels: true

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, memory_limiter, resourcedetection]
      exporters: [googlecloud]
    traces:
      receivers: [otlp]
      processors: [batch, memory_limiter, resourcedetection]
      exporters: [googlecloud]

Step 2: Configure Proxy Adaptor

Follow these steps to enable and configure OTEL in your application:

1. Edit the config.yaml (or set up the CONFIG_FILE environment variable) File:

   • Set the enabled field to True to enable metrics and traces.
   • Configure the endpoints for the collector service to export the metrics and traces.
   • Set the healthcheck endpoint, which is configured in the collector service. Avoid including http:// in the endpoints. Refer to the example_config.yaml file for guidance.

2. Example Configuration Block for OTEL:

      # Enable the health check in this proxy application config only if the
      # "health_check" extension is added to the OTEL collector service configuration.
      #
      # Recommendation:
      # Enable the OTEL health check if you need to verify the collector's availability
      # at the start of the application. For development or testing environments, it can
      # be safely disabled to reduce complexity.
   otel:
       # Set enabled to true or false for OTEL metrics and traces
       enabled: True
       # Name of the collector service to be setup as a sidecar
       serviceName: YOUR_OTEL_COLLECTOR_SERVICE_NAME
       healthcheck:
           # Enable/Disable Health Check for OTEL, Default 'False'.
           enabled: False
           # Health check endpoint for the OTEL collector service
           endpoint: YOUR_OTEL_COLLECTOR_HEALTHCHECK_ENDPOINT
       metrics:
           # Collector service endpoint
           endpoint: YOUR_OTEL_COLLECTOR_SERVICE_ENDPOINT
       traces:
           # Collector service endpoint
           endpoint: YOUR_OTEL_COLLECTOR_SERVICE_ENDPOINT
           # Sampling ratio should be between 0 and 1.
           samplingRatio: YOUR_SAMPLING_RATIO
   
   
   

Step 3: Setup Proxy Adaptor & OTEL Collector service as a sidecar on GKE.

1. Use the proxy-adapter-application-as-sidecar.yaml file from the deployment/sidecar-k8 to setup the otel collector service along with proxy adaptor as a sidecar. You can find step by step instructions in the /deployment/sidecar-k8/README.md file.

Step 4: Verify OTEL changes

1. Verify the Health Check:

Run a curl command from the k8 pod to http://collector-ip:13133/health_check. You should see a health status message.

1. Check OTEL traces on GCP

• Login to your GCP project and open Monitoring service.
• Open the sidebar on the left and click on Trace Explorer
• You should be able to see the traces as shown below:

1. Check OTEL metrics on GCP

• Login to your GCP project and open Monitoring service.
• Open the sidebar on the left and click on Metrics Explorer
• You should be able to see the categories as shown below:

• Select options Prometheus Target/Spanner
• To view the metrics for total number of requests, select option prometheus/spanner_cassandra_adapter_<dbname>_request_count_total/counter

• To view the metrics for latency, select option prometheus/spanner_cassandra_adapter_round_trip_latencies_milliseconds/histogram

• You can also view other metrics related to spanner library under the Prometheus Target/Spanner category.

Documentation

Index

• func AddAnnotation(ctx context.Context, event string)
• func AddAnnotationWithAttr(ctx context.Context, event string, attr []attribute.KeyValue)
• type Attributes
• type MeterProvider
• type OTelConfig
• type OpenTelemetry
  • func NewOpenTelemetry(ctx context.Context, config *OTelConfig, logger *zap.Logger) (*OpenTelemetry, func(context.Context) error, error)
  • func (o *OpenTelemetry) EndSpan(span trace.Span)
  • func (o *OpenTelemetry) InitMeterProvider(ctx context.Context, resource *resource.Resource) (*sdkmetric.MeterProvider, error)
  • func (o *OpenTelemetry) InitTracerProvider(ctx context.Context, resource *resource.Resource) (*sdktrace.TracerProvider, error)
  • func (o *OpenTelemetry) RecordError(span trace.Span, err error)
  • func (o *OpenTelemetry) RecordLatencyMetric(ctx context.Context, duration time.Time, attrs Attributes)
  • func (o *OpenTelemetry) RecordRequestCountMetric(ctx context.Context, attrs Attributes)
  • func (o *OpenTelemetry) SetError(ctx context.Context, err error)
  • func (o *OpenTelemetry) StartSpan(ctx context.Context, name string, attrs []attribute.KeyValue) (context.Context, trace.Span)
• type TelemetryInitializer
• type TracerProvider

Functions

func AddAnnotation

func AddAnnotation(ctx context.Context, event string)

AddAnnotation add event to the span of the given ctx.

func AddAnnotationWithAttr

func AddAnnotationWithAttr(ctx context.Context, event string, attr []attribute.KeyValue)

AddAnnotationWithAttr add event to the span of the given ctx with the necessary attributes.

Types

type Attributes

type Attributes struct {
	Method    string
	Status    string
	QueryType string
}

type MeterProvider

type MeterProvider interface {
	InitMeterProvider(ctx context.Context) (*sdkmetric.MeterProvider, error)
}

MeterProvider defines the interface for creating meters.

type OTelConfig

type OTelConfig struct {
	TracerEndpoint     string
	MetricEndpoint     string
	ServiceName        string
	TraceSampleRatio   float64
	OTELEnabled        bool
	Database           string
	Instance           string
	HealthCheckEnabled bool
	HealthCheckEp      string
	ServiceVersion     string
}

OTelConfig holds configuration for OpenTelemetry.

type OpenTelemetry

type OpenTelemetry struct {
	Config         *OTelConfig
	TracerProvider *sdktrace.TracerProvider
	MeterProvider  *sdkmetric.MeterProvider
	Tracer         trace.Tracer
	Meter          metric.Meter

	Logger *zap.Logger
	// contains filtered or unexported fields
}

OpenTelemetry provides methods to setup tracing and metrics.

func NewOpenTelemetry

func NewOpenTelemetry(ctx context.Context, config *OTelConfig, logger *zap.Logger) (*OpenTelemetry, func(context.Context) error, error)

NewOpenTelemetry creates and initializes a new instance of OpenTelemetry, including its Tracer and Meter providers, and returns Tracer and Meter instances.

func (*OpenTelemetry) EndSpan

func (o *OpenTelemetry) EndSpan(span trace.Span)

EndSpan stops the span.

func (*OpenTelemetry) InitMeterProvider

func (o *OpenTelemetry) InitMeterProvider(ctx context.Context, resource *resource.Resource) (*sdkmetric.MeterProvider, error)

InitMeterProvider initializes the MeterProvider for OpenTelemetry. This function sets up a gRPC exporter for metrics data, targeting the configured MetricEndpoint. It returns an initialized MeterProvider or an error if the setup fails. The MeterProvider is responsible for collecting and exporting metrics from your application to an OpenTelemetry Collector or directly to a backend that supports OTLP over gRPC for metrics.

func (*OpenTelemetry) InitTracerProvider

func (o *OpenTelemetry) InitTracerProvider(ctx context.Context, resource *resource.Resource) (*sdktrace.TracerProvider, error)

InitTracerProvider initializes the TracerProvider for OpenTelemetry. This function configures a gRPC exporter for trace data, pointing to the configured TracerEndpoint. It returns an initialized TracerProvider or an error if the initialization fails.

func (*OpenTelemetry) RecordError

func (o *OpenTelemetry) RecordError(span trace.Span, err error)

RecordError records a new error under a span.

func (*OpenTelemetry) RecordLatencyMetric

func (o *OpenTelemetry) RecordLatencyMetric(ctx context.Context, duration time.Time, attrs Attributes)

RecordLatencyMetric adds the latency metric based on provided context, name, duration and attributes.

func (*OpenTelemetry) RecordRequestCountMetric

func (o *OpenTelemetry) RecordRequestCountMetric(ctx context.Context, attrs Attributes)

RecordRequestCountMetric adds the request count based on provided context, name and attributes.

func (*OpenTelemetry) SetError

func (o *OpenTelemetry) SetError(ctx context.Context, err error)

SetError records an error for the span retrieved from the provided context. If OpenTelemetry (OTEL) is not enabled or if the error is nil, the function will return immediately without recording the error. If enabled and an error is present, the error will be recorded and the span status will be set to error with the corresponding error message.

Parameters: - ctx: The context from which the span is retrieved. - err: The error to be recorded and set on the span.

func (*OpenTelemetry) StartSpan

func (o *OpenTelemetry) StartSpan(ctx context.Context, name string, attrs []attribute.KeyValue) (context.Context, trace.Span)

CreateTrace starts a new trace span based on provided context, name, attributes, and error. It returns a new context containing the span.

type TelemetryInitializer

type TelemetryInitializer interface {
	InitOpenTelemetry(ctx context.Context) (shutdown func(context.Context) error, err error)
}

TelemetryInitializer defines the interface for initializing OpenTelemetry components.

type TracerProvider

type TracerProvider interface {
	InitTracerProvider(ctx context.Context) (*sdktrace.TracerProvider, error)
}

TracerProvider defines the interface for creating traces.