lake

README

Dev Lake

English	中文

What is Dev Lake?

Dev Lake is the one-stop solution that integrates, analyzes, and visualizes software development data throughout the software development life cycle (SDLC) for engineering teams.

Dashboard Screenshot

User Flow

Why Dev Lake?

1. Visualize and analyze your entire SDLC process in one personalized, unified view.
2. Debug process- and team-level issues based on unified and standardized measures.

What can be accomplished with Dev Lake?

1. Unifies data from multiple sources (Jira, GitLab, Github, Jenkins, etc.) in one place to break data isolation.
2. See a series of industry-standard metrics to identify engineering problems. For example, long requirement lead time, high bug rate per 1k lines of code, etc.
3. Highly customizable charts, metrics & dashboards for users to find insights based on their own needs.

Contents

Section	Subsection	Description	Documentation Link
Data Sources	Data Sources	Links to specific plugin usage & details	View Section
Setup Guide	User Setup	Set up Dev Lake locally as a user	View Section
	Developer Setup	Set up development environment locally	View Section
	Cloud Setup	Set up DevLake in the cloud with Tin	View Section
Tests	Tests	Commands for running tests	View Section
Make Contribution	Build a Plugin	Details on how to make your own plugin	View Section
	Add Plugin Metrics	Guide to add metrics	View Section
	Contribution specs	How to contribute to this repo	View Section
User Guide, Help, and more	Grafana	How to visualize the data	View Section
	Need Help	Message us on Discord	View Section
	FAQ	Frequently asked questions by users	View Section
	License	The project license	View Section

Data Sources We Currently Support

Below is a list of data source plugins used to collect & enrich data from specific sources. Each has a README.md file with basic setup, troubleshooting, and metrics info.

For more information on building a new data source plugin, see Build a Plugin.

Section	Section Info	Docs
Jira	Summary, Data & Metrics, Configuration, Plugin API	Link
GitLab	Summary, Data & Metrics, Configuration, Plugin API	Link
Jenkins	Summary, Data & Metrics, Configuration, Plugin API	Link
GitHub	Summary, Data & Metrics, Configuration, Plugin API	Link

Setup Guide

There're 3 ways to set up Dev Lake.

User setup

NOTES:

• If you only plan to run the product locally, this is the only section you should need.
• Commands written like this are to be run in your terminal.

Required Packages to Install

• Docker
• docker-compose

NOTE: After installing docker, you may need to run the docker application and restart your terminal

Commands to run in your terminal

1. Clone repository:

   git clone https://github.com/merico-dev/lake.git devlake
   cd devlake
   cp .env.example .env
   

2. Start Docker on your machine, then run docker-compose up -d to start the services.

3. Visit localhost:4000 to setup configuration files.

   • Navigate to desired plugins pages on the Integrations page
   • You will need to enter the required information for the plugins you intend to use.
   • Please reference the following for more details on how to configure each one: -> Jira -> GitLab -> Jenkins -> GitHub

   • Submit the form to update the values by clicking on the Save Connection button on each form page

   • devlake takes a while to fully boot up. if config-ui complaining about api being unreachable, please wait a few seconds and try refreshing the page.
   • To collect this repo for a quick preview, you must put your github into Auth Token(s) Data Integrations / Github page.

4. Visit localhost:4000/triggers to trigger data collection.

   • Please refer to this wiki How to trigger data collection. Data collection can take up to 20 minutes for large projects. (GitLab 10k+ commits or Jira 5k+ issues)
   • To collect this repo for a quick preview, you can use the following JSON

     [
       [
         {
           "Plugin": "github",
           "Options": {
             "repositoryName": "lake",
             "owner": "merico-dev"
           }
         }
       ],
       [
         {
           "plugin": "github-domain",
           "options": {}
         }
       ]
     ]
     

5. Click View Dashboards button when done (username: admin, password: admin). The button will be shown on the Trigger Collection page when data collection has finished.

Setup cron job

Commonly, we have the requirement to synchronize data periodically. We provided a tool called lake-cli to meet that requirement. Check lake-cli usage here.

Otherwise, if you just want to use the cron job, please check docker-compose version at here

Developer Setup

Requirements

• Docker
• Golang
• Make
  • Mac (Already installed)
  • Windows: Download
  • Ubuntu: sudo apt-get install build-essential

How to setup dev environment

1. Navigate to where you would like to install this project and clone the repository:

   git clone https://github.com/merico-dev/lake.git
   cd lake
   

2. Install Go packages:

   make install
   

3. Copy the sample config file to new local file:

   cp .env.example .env
   

   Find the line start with DB_URL in the file .env and replace mysql:3306 with 127.0.0.1:3306

4. Start the Docker containers:

   Make sure the Docker daemon is running before this step.

   make compose
   

5. Run the project:

   make dev
   

6. Visit localhost:4000 to setup configuration files.

   • Navigate to desired plugins pages on the Integrations page
   • You will need to enter the required information for the plugins you intend to use.
   • Please reference the following for more details on how to configure each one: -> Jira -> GitLab, -> Jenkins -> GitHub

   • Submit the form to update the values by clicking on the Save Connection button on each form page

7. Visit localhost:4000/triggers to trigger data collection.

   • Please refer to this wiki How to trigger data collection. Data collection can take up to 20 minutes for large projects. (GitLab 10k+ commits or Jira 5k+ issues)

8. Click View Dashboards button when done (username: admin, password: admin). The button is shown in the top left.

Cloud setup

If you want to run Dev Lake in a clound environment, you can set up Dev Lake with Tin. See detailed setup guide

Disclaimer:

To protect your information, it is critical for users of the Tin hosting to set passwords to protect Dev Lake applications. We built Dev Lake as a self-hosted product, in part to ensure users have total protection and ownership of their data, while the same remains true for the Tin hosting, this risk point can only be eliminated by the end-user.

Tests

To run the tests:

make test

Make Contribution

This section list all the documents to help you contribute to the repo.

Add a Plugin

plugins/README.md

Add Plugin Metrics

plugins/HOW-TO-ADD-METRICS.md

Contributing Spec

CONTRIBUTING.md

User Guide, Help and more

Grafana

We use Grafana as a visualization tool to build charts for the data stored in our database. Using SQL queries, we can add panels to build, save, and edit customized dashboards.

All the details on provisioning and customizing a dashboard can be found in the Grafana Doc.

Need help?

Message us on Discord

FAQ

Q: When I run docker-compose up -d I get this error: "qemu: uncaught target signal 11 (Segmentation fault) - core dumped". How do I fix this?

A: M1 Mac users need to download a specific version of docker on their machine. You can find it here.

License

This project is licensed under Apache License 2.0 - see the LICENSE file for details.