datasource-syncer

README

Data Source Syncer

This CLI tool acts as a cron job which remotely syncs data to a given Grafana Prometheus data source. This ensures that the Grafana data source has the following set correctly:

• The Prometheus server URL
• GET HTTP method
• The Prometheus type and version
• Authentication by refreshing a oAuth2 access token periodically

By regularly refreshing the oAuth2 access token, you can configure Grafana to directly query Google Cloud Monitoring (Managed Service for Prometheus).

Google access tokens have a lifetime of 1 hour. This script runs every 30 minutes to ensure you have an uninterrupted connection between Grafana and Google Cloud Monitoring.

Run

1. Figure out the URL of your Grafana instance. e.g. https://your.grafana.net for a Grafana Cloud instance or http://localhost:3000 for a local instance.

2. Choose any pre-existing Grafana Prometheus data source that you would like to use for Managed Service for Prometheus, or create a new Grafana Prometheus data source and save it. Once that's done, find the data source UID. The data source UID is found in the URL when configuring or exploring a data source. The data source UID is the last part of the URL, when configuring a data source, e.g. https://your.grafana.net/connections/datasources/edit/<datasource_uid>.

3. Set up a Grafana service account and generate a token. Set the expiration date to never.

Kubernetes CronJob

Set up the following environment variables:

# These values are required.
DATASOURCE_UIDS=YOUR_DATASOURCE_UIDs
GRAFANA_API_TOKEN=YOUR_GRAFANA_SERVICE_ACCOUNT_TOKEN
GRAFANA_API_ENDPOINT=YOUR_GRAFANA_INSTANCE_URL
PROJECT_ID=PROJECT_ID_TO_QUERY_GCM
 # Optional Credentials file. Can be left empty if default credentials have sufficient permission.
GOOGLE_APPLICATION_CREDENTIALS=OPTIONAL_GOOGLE_CLOUD_SERVICE_ACCOUNT_WITH_GOOGLE_CLOUD_MONITORING_READ_ACCESS

Running the following Cron job will refresh the data source on initialization and every 30 minutes:

cat datasource-syncer.yaml \
| sed 's|$DATASOURCE_UIDS|'"$DATASOURCE_UIDS"'|; s|$GRAFANA_API_ENDPOINT|'"$GRAFANA_API_ENDPOINT"'|; s|$GRAFANA_API_TOKEN|'"$GRAFANA_API_TOKEN"'|; s|$PROJECT_ID|'"$PROJECT_ID"'|;' \
| kubectl apply -f -

Query Across Multiple Projects

To query across multiple projects, you must create a metrics scope and authorize the local project's default compute service account to have monitoring.read access to the scoping project. If your local project is your scoping project, then this permission is granted by default and cross-project querying should work with no further configuration.

Workload Identity Setup

If you're using WLI you need to grant the service account these two permissions: Monitoring Viewer and Service Account Token Creator.

Development

go run main.go \
  --query.credentials-file=$GOOGLE_APPLICATION_CREDENTIALS \
  --datasource-uids=$DATASOURCE_UIDS \
  --grafana-api-token=$GRAFANA_API_TOKEN \
  --grafana-api-endpoint=$GRAFANA_API_ENDPOINT \
  --project-id=$PROJECT_ID