airbyte_exporter

README

Airbyte Prometheus Exporter

Metrics exposed

Metric	Type	Labels
airbyte_jobs_completed_total	Counter	destination_connector, source_connector, schedule_type, type, status
airbyte_connections	Gauge	destination_connector, source_connector, status
airbyte_sources	Gauge	source_connector, tombstone
airbyte_destinations	Gauge	destination_connector, tombstone
airbyte_jobs_pending	Gauge	destination_connector, source_connector, schedule_type, type
airbyte_jobs_running	Gauge	destination_connector, source_connector, schedule_type, type
airbyte_connections_last_successful_sync_age_hours	Histogram	destination_connector, source_connector, schedule_type

Configuration

airbyte_exporter can be configured via:

• environment variables, e.g. AIRBYTE_EXPORTER_DB_PASSWORD=p455w0rd
• a configuration file
• POSIX flags, e.g. --db-password p455w0rd

Available flags can be listed using the program's help:

$ ./airbyte_exporter --help

Airbyte Exporter

Usage:
  airbyte_exporter [flags]

Flags:
      --db-addr string       Database address (host:port) (default "localhost:5432")
      --db-name string       Database name (default "airbyte")
      --db-password string   Database password (default "airbyte_exporter")
      --db-sslmode string    Database sslmode (default "disable")
      --db-user string       Database user (default "airbyte_exporter")
  -h, --help                 help for airbyte_exporter
      --listen-addr string   Listen to this address (host:port) (default "0.0.0.0:8080")
      --log-level string     Log level (trace, debug, info, warn, error, fatal, panic) (default "info")

Example configuration file

The exporter will look for configuration files located under: - /etc/airbyte_exporter.yaml - ~/.config/airbyte_exporter.yaml

# Global exporter options
listen-addr: 0.0.0.0:8080
log-level: info

# Airbyte database options
db-addr: "postgresql:5432"
db-name: airbyte
db-password: "ch4ng3m3!"
db-sslmode: require

PostgreSQL user

The exporter needs to be able to connect to the Airbyte database, and have read-only access to Airbyte database tables.

The following commands are provided as an example; see PostgreSQL's documentation for further information:

• CREATE ROLE
• GRANT
• Predefined roles

Connect to the airbyte database:

# psql -h <host> -p <port> -U <admin_user> <database>
$ psql -h 127.0.0.1 -p 5432 -U postgres airbyte

Create the user:

CREATE ROLE airbyte_exporter WITH LOGIN ENCRYPTED PASSWORD 'SomeStrongPassword';

For PostgreSQL version 14 and above, grant read-only privileges with:

GRANT pg_read_all_data TO airbyte_exporter;

For PostgreSQL version 13 and below, grant read-only privileges on current and newly created tables with:

-- Current tables
GRANT CONNECT ON DATABASE YourDatabaseName TO airbyte_exporter;
GRANT USAGE ON SCHEMA public TO airbyte_exporter;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO airbyte_exporter;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO airbyte_exporter;
REVOKE CREATE ON SCHEMA public FROM PUBLIC;

-- Newly created tables
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO airbyte_exporter;

Running

Running with Docker

See airbyte_exporter container packages for a list of available Docker image tags.

Pull the Docker image:

$ docker pull ghcr.io/botify-labs/airbyte_exporter:latest

Run the exporter:

$ docker run \
    --name airbyte-exporter \
    --rm \
    -e AIRBYTE_EXPORTER_DB_HOST=postgresql \
    -e AIRBYTE_EXPORTER_DB_PASSWORD=ch4ng3m3 \
    -p 8080:8080
    ghcr.io/botify-labs/airbyte_exporter:latest

Deploying to Kubernetes with Helm

See instructions on Artifact Hub for botify-helm-charts/prometheus-airbyte-exporter.

Change Log

See CHANGELOG

Contributing

See CONTRIBUTING

License

airbyte_exporter is licensed under the MIT License.