gitaly-blackbox

README

gitaly-blackbox

Gitaly-blackbox is a Prometheus exporter that measures GitLab server performance by performing a Git HTTP clone of one or more given repositories.

The intended application is to deploy gitaly-blackbox in some fixed location and to point it at a list of Git HTTP URL's on your GitLab server. Gitaly-blackbox will then periodically perform a "fake clone" that exercises the server side of the Git transport protocol, without needing local disk space to write the cloned repository. After the fake clone finishes, gitaly-blackbox updates a set of Prometheus gauges that indicates measurements such as the size of the clone in bytes, the time to the first progress message, the time to the first packet of packfile data, etc.

You can then set up Prometheus dashboards that track the clone performance of the given repos over time, in a way that correlates with the user experience of cloning the repos.

Caveat: gitaly-blackbox analyzes Git clones at the Git transport level. It does not verify or analyze the actual Git objects returned by the server.

Configuration

Also see config.toml.example.

Global settings

Settings at the top of the config file are global.

Setting	Type	Default	Required?	Description
prometheus_listen_addr	string	none	yes	Network address to open the prometheus listener on, e.g. 0.0.0.0:9687
sleep	int	900	no	Sleep time in between probes, in seconds. Use this to tune how many probes you run vs how much strain you put on your GitLab server.

Logging settings

Settings under [logging], but before the probes, constitute logging settings.

Setting	Type	Default	Required?	Description
level	string	'info'	no	Log level (error, warn, info, debug etc.)
format	string	'text'	no	Log format: text or json.

Probe settings

Probes are defined by a [[probe]] heading followed by key value pairs.

Setting	Type	Default	Required?	Description
name	string	none	yes	Probe name. This must be unique. It will show up in Prometheus as a label value.
url	string	none	yes	HTTP or HTTPS Git clone URL, such as https://gitlab.com/gitlab-org/gitlab-test.git
user	string	none	no	HTTP Basic username
password	string	none	no	HTTP Basic password

Metrics

Gitaly-blackbox exports a number of metrics for each probe defined in the config file. A Git HTTP clone consists of two HTTP requests: a GET followed by a POST. Most metrics are specific to either the GET or the POST.

gitaly_blackbox_git_http_get_first_packet_seconds

The time from the start of the GET request to the first Git transport packet in the response. This is an indication of how long it took the server to prepare generating the response.

gitaly_blackbox_git_http_get_total_time_seconds

The total time to finish the GET request. This includes the time needed to receive the response body.

gitaly_blackbox_git_http_get_advertised_refs

The main purpose of the GET request in Git HTTP is to provide the client with a list of refs (branches and tags) it may clone. This metric records the number of refs advertised by the server.

gitaly_blackbox_git_http_wanted_refs

Not all refs advertised by the server will be selected during a normal clone. For example, GitLab lets users fetch refs that correspond to merged Merge Requests, for which the original branch got deleted. Such "extra" refs are excluded from a regular git clone and this is also what gitaly-blackbox does. This metric records how many of the refs advertised by the server belong in a regular clone.

gitaly_blackbox_git_http_post_total_time_seconds

This is the total time to finish the POST request of the clone. This includes downloading the pack data. Note that during a normal clone, Git will also spend time verifying the data that was sent back, and writing a working directory to disk. That means that from a user's point of view, git clone runs for longer than the number we are measuring here.

gitaly_blackbox_git_http_post_first_progress_packet_seconds

A Git HTTP clone uses a "multiband" stream of Git transport packets. There are three bands: pack, progress and error. During a clone, the server will start sending progress information before it has finished gathering the contents of the clone. This metric measures the time from the start of the POST to the first progress packet returned by the server.

The first progress message is usually:

remote: Enumerating objects: ...

gitaly_blackbox_git_http_post_first_pack_packet_seconds

This metric measures the time from the start of the POST to the first pack packet returned by the server. This is an indication of how long it took the server to decide what data to include in the clone.

On the client side, this is the moment you start seeing:

Receiving objects: ...

gitaly_blackbox_git_http_post_pack_bytes

This metric is the total amount of pack data returned by the server. This correlates with the size of the clone on the client side. Note that an actual clone will be bigger on the client's disk because the client computes a pack index file based on the pack data.

This metric excludes progress messages; it measures the pack band only.