liatrio-otel-collector
The Liatrio OTEL Collector is an upstream distribution of the Open Telemetry
collector, rebuilt with custom packages hosted within this repository. These
custom packages are by default targeted for downstream contribution to Open
Telemetry; pursuant acceptance by the community.
Quick Start Guide
Before diving into the codebase, you'll need to set up a few things. This guide
is designed to help you get up and running in no time!
Here are the steps to quickly get started with this project:
-
Install Go: Use Homebrew to install Go with the following command:
brew install go
-
Install pre-commit: With the help of Homebrew, install the pre-commit as
follows:
brew install pre-commit
-
Initialize pre-commit: Once installed, initiate pre-commit by running:
pre-commit install
-
Set up GitHub and/or GitLab scraper(s).
-
Start the collector: Finally, initiate the program:
make run
To configure the GitHub Scraper you will need to make the following changes to
config/config.yaml:
-
Uncomment/add extensions.basicauth/github
section
basicauth/github:
client_auth:
username: ${env:GITHUB_USER}
password: ${env:GITHUB_PAT}
-
Uncomment/add receivers.gitprovider.scrapers.github.auth
section
auth:
authenticator: basicauth/github
-
Add basicauth/github
to the service.extensions
list
extensions: [health_check, pprof, zpages, basicauth/github]
-
Set environment variables: GITHUB_ORG
, GITHUB_USER
, and GITHUB_PAT
To configure the GitLab Scraper you will need to make the following changes to
config/config.yaml
-
Uncomment/add extensions.bearertokenauth/gitlab
section
bearertokenauth/gitlab:
token: ${env:GITLAB_PAT}
-
Uncomment/add receivers.gitprovider.scrapers.gitlab.auth
section
auth:
authenticator: bearertokenauth/gitlab
-
Add bearertokenauth/gitlab
to the service.extensions
list
extensions: [health_check, pprof, zpages, bearertokenauth/gitlab]
-
Set environment variables: GL_ORG
and GITLAB_PAT
Exporting to Grafana Cloud
If you want to export your data to Grafana Cloud through their OTLP endpoint,
there's a couple of extra things you'll need to do.
- Run
export GRAF_USER
and export GRAF_PAT
with your instance id and cloud
api key
- Update the config/config.yaml file with the following:
extensions:
...
basicauth/grafana:
client_auth:
username: ${env:GRAF_USER}
password: ${env:GRAF_PAT}
...
exporters:
...
otlphttp:
auth:
authenticator: basicauth/grafana
endpoint: https://otlp-gateway-prod-us-central-0.grafana.net/otlp
...
service:
extensions: [..., basicauth/grafana]
pipelines:
metrics:
receivers: [otlp, gitprovider]
processors: []
exporters: [..., otlphttp]
Debugging
To debug through vscode
:
-
Create a .debug.env
file in the root of the repo, make a copy of the
example
-
run make build-debug
-
run cd build && code .
-
With your cursor focused in main.go
within the build
directory. Launch
the Launch Otel Collector in debug"
debug configuration.
OTEL Intro
OTEL is a protocol used for distributed logging, tracing, and metrics.
To collect metrics from various services, we need to configure receivers.
OTEL provides many built-in receivers, but in certain cases, we may need to
create custom receivers to meet our specific requirements.
A custom receiver in OTEL is a program that listens to a specific endpoint and
receives incoming log, metrics, or trace data. This receiver then pipes the
content to a process, for it to then be exported to a data backend.
Creating a custom receiver in OTEL involves implementing the receiver interface
and defining the endpoint where the receiver will listen for incoming trace data.
Once the receiver is implemented, it can be deployed to a specific location and
configured to start receiving trace data.
Prereqs
There is currently a guide to build a custom trace receiver. It is a long read,
requires a fairly deep understanding, and is slightly out of date due to
non-backwards compatible internal API breaking changes. This document and
receiver example attempts to simplify that to an extent.
There are a few main concepts that should help you get started:
- Get familiar with the
ocb
tool. It is used to build custom collectors using
a build-config.yaml
file.
- Get familiar with
Go
& the Factory
design pattern.
- Clearly define what outcome you want before building a customization.
- Get familiar with
Go interfaces
.
- Get familiar with
delv
the go debugger.
References & Useful Resources