fuse

README

fuse

beta

MergeStat fuse is a project to sync data from mergestat-lite queries into PostgreSQL tables. It behaves similar to an ETL tool, taking data from git repositories and the GitHub API and regularly updating tables in a database. It's intended purpose is to enable "operational analytics for engineering teams," where downstream tools may consume the data from the PostgreSQL database.

Running Locally

Try Fuse out locally with docker-compose by cloning this repository and running

docker-compose up

Now if you visit http://localhost:3300/ you should be able to access our management UI where you can begin adding repositories and syncing data.

NOTE if you'd like to make use of GitHub API data (any sync type that uses the GitHub API including repo auto imports) or any private GitHub repos you will need to supply a GitHub personal access token. We are working on GitHub auth alternatives, but for now a PAT is your best bet.

You can manage a single PAT for your instance in the Settings area of the management UI.

Examples

Take a look at all of our examples

You can also visit http://localhost:3000/ to access a local Grafana instance with example dashboards using the data from Fuse 🎉 :

Resetting an Instance

If you'd like to "start from scratch" (i.e. reset the Fuse DB and start with a fresh deployment), run the following:

docker-compose down

docker volume rm fuse_db_data

Data Types

• GIT_COMMITS Retrieves the commit history of a repo
• GIT_COMMIT_STATS Retrieves commit stats for a repo
• GIT_FILES Retrieves files (content and paths) of a git repo
• GITHUB_PR_COMMITS Retrieves commits for all pull requests in a GitHub repo
• GITHUB_PR_REVIEWS Retrieves the reviews of all pull requests in a GitHub repo
• GITHUB_REPO_ISSUES Retrieves all the issues of a GitHub repo
• GITHUB_REPO_METADATA Retrieves metadata about a GitHub repo
• GITHUB_REPO_PRS Retrieves all the pull requests of a GitHub repo
• GITHUB_REPO_STARS Retrieves all stargazers of a GitHub repo
• GIT_REFS Retrieves all the refs of a git repo
• TRIVY_REPO_SCAN Executes a trivy scan on a git repository

Database Migrations

Running & Testing Locally

We use this tool https://github.com/golang-migrate/migrate/tree/master/cmd/migrate to handle migrations. In order to generate new migrations, use the migrate CLI like so:

migrate create -ext sql -dir migrations/ your_migration_subject

You can then run docker-compose up (without -d) so that you can check in the logs if the migration is successful.

Dealing with Dirty Migrations

• Stop the docker container with all the images but the postgress one.
• Then use this command migrate -path ./migrations -database "postgres://postgres:password@localhost:5432/postgres?sslmode=disable" force {last-successful-version}.
• After that delete the up and down of your failed migration and recreate another migration by migrate create -ext sql -dir migrations/ you_migration_subject.
• And you should be good to go.

Running on Windows

To run fuse on Windows you will need to use WSL (Windows subsystem for linux) and modify settings in the docker desktop application.

• Modify your current docker desktop settings to ones showed in the pictures.

• Follow the instructions provided here to make correct use of WSL
• Try to run docker-compose up inside the WSL

SQL Linting

The .sql files in this repo are linted using sqlfluff and .sqlfluff configuration file at the root of this repo.

To check for linting issues run this command from the root of the root directory

sqlfluff lint

To force linting issue fixes run this command from the root of the root directory

sqlfluff fix