overcover

README

=========
Overcover
=========

.. image:: https://img.shields.io/github/tag/klmitch/overcover.svg
    :target: https://github.com/klmitch/overcover/tags
.. image:: https://img.shields.io/hexpm/l/plug.svg
    :target: https://github.com/klmitch/overcover/blob/master/LICENSE
.. image:: https://travis-ci.org/klmitch/overcover.svg?branch=master
    :target: https://travis-ci.org/klmitch/overcover
.. image:: https://coveralls.io/repos/github/klmitch/overcover/badge.svg?branch=master
    :target: https://coveralls.io/github/klmitch/overcover?branch=master
.. image:: https://godoc.org/github.com/klmitch/overcover?status.svg
    :target: http://godoc.org/github.com/klmitch/overcover
.. image:: https://img.shields.io/github/issues/klmitch/overcover.svg
    :target: https://github.com/klmitch/overcover/issues
.. image:: https://img.shields.io/github/issues-pr/klmitch/overcover.svg
    :target: https://github.com/klmitch/overcover/pulls
.. image:: https://goreportcard.com/badge/github.com/klmitch/overcover
    :target: https://goreportcard.com/report/github.com/klmitch/overcover

This repository contains Overcover.  Overcover is a testing tool meant
to work in conjunction with the coverage profile data generated by the
``go test`` tool with the ``-coverprofile`` option; it computes an
overall coverage of the entire source base and emits that
information.  Overcover can also optionally enforce a minimum coverage
threshold, allowing test jobs to be failed if the testing coverage has
been weakened.

Basic Usage
===========

To use Overcover, install and execute it like so::

    % overcover --coverprofile coverage.out

This will read the indicated coverage profile and emit a message
describing the overall coverage represented by the profile.  A
threshold can be enforced, which will cause Overcover to exit with a
status code of 1 if the coverage does not reach the threshold::

    % overcover --coverprofile coverage.out --threshold 75

These values can also be passed to Overcover using environment
variables named after the long option names--e.g.,
``OVERCOVER_COVERPROFILE`` specifies a value for ``--coverprofile``,
and ``OVERCOVER_THRESHOLD`` specifies a value for ``--threshold``.

The ``go test`` command does not include packages that do not contain
any test files in the output coverage profile.  To ensure that all
code is counted in the coverage determination, Overcover can be
provided the package specification in the same format taken by ``go
test``; it will count the number of statements and include that data
in its calculation.  For example::

    % overcover --coverprofile coverage.out ./...

Build arguments may be provided to influence the selection of source
packages; these are standard build arguments to the ``go`` tools, and
are specified with a ``--build-arg`` prefix; the corresponding
environment variable, ``OVERCOVER_BUILD_ARG``, should contain a
space-separated sequence of build arguments.

Finally, summary information on the coverage per package can be
emitted using ``--summary`` (``OVERCOVER_SUMMARY``), and detailed
per-file information can be emitted using ``--detailed``
(``OVERCOVER_DETAILED``); these two options may be used together, in
which case the package data will be emitted prior to the file data.
The output is sorted first to place the lowest coverage at the top,
then it is sorted lexically by package or file name.

Configuration File
==================

Overcover can also read a configuration file containing the target
threshold, which may be in any number of formats as long as the file
has a proper extension; e.g., for a YAML file, the configuration file
name should end with ".yaml" or ".yml".  The values in this file are
named the same as the command line options, and the configuration file
may be specified to Overcover using either ``OVERCOVER_CONFIG`` or the
``--config`` command line option::

    % overcover --config config.yaml

An example of that file could be as follows::

    ---
    threshold: 75

Automatically Updating the Threshold
------------------------------------

When using a configuration file with Overcover, the Overcover tool can
automatically update the threshold in the file, to help ensure that
coverage can never decrease.  This is done using the *headroom*
options, ``--min-headroom`` and ``--max-headroom``
(``OVERCOVER_MIN_HEADROOM`` and ``OVERCOVER_MAX_HEADROOM`` for
environment variables, and ``min_headroom`` and ``max_headroom`` in
the configuration files).  If the total coverage exceeds the threshold
plus the maximum headroom, then the threshold will be updated to be
the total coverage minus the minimum headroom.  An example of a
configuration file set up to use this automatic update feature would
be as follows::

    ---
    threshold: 75
    min_headroom: 1
    max_headroom: 2

To provide a concrete example of the automatic threshold update, let's
assume that we're using the above configuration file, and that the
actual coverage is 80%.  Since ``max_headroom`` is set to 2, that
means that the coverage would be greater than ``threshold +
max_headroom`` (77%), so the threshold would be updated to the
coverage minus the ``min_headroom``, or 79%, and this configuration
file would become::

    ---
    threshold: 79
    min_headroom: 1
    max_headroom: 2

(Note that the configuration file after the update likely won't look
*exactly* like this, as it is programmatically generated.)

Automatic threshold update only occurs when a configuration file is
specified using ``--config`` or the environment variable
``OVERCOVER_CONFIG``, and when ``max_headroom`` is set to a value
strictly greater than ``min_headroom``.  Defaults for all these values
are shown below.

Preventing Automatic Update
---------------------------

Sometimes it is convenient to prevent an automatic update of the
configuration file, such as during a CI run.  This can be done by
using the ``--readonly`` flag, or setting the ``OVERCOVER_READONLY``
environment variable.  (The variable need not have any particular
value; it need only appear in the environment.)  In this scenario,
Overcover will exit with a status code of 5 to indicate that the
configuration file needs to be updated with a new threshold; it will
also emit a human-readable error message to the standard error stream
indicating what the threshold should be set to.

Options/Configuration Table
===========================

+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
| Configuration | Environment Variable   | Command Line Option | Default    | Description                                                              |
+===============+========================+=====================+============+==========================================================================+
|               | OVERCOVER_COVERPROFILE | --coverprofile (-p) | *Required* | Name of the coverage profile generated by ``go test``.                   |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
| threshold     | OVERCOVER_THRESHOLD    | --threshold (-t)    | 0.0        | Minimum coverage threshold required.                                     |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
| min_headroom  | OVERCOVER_MIN_HEADROOM | --min-headroom (-m) | 0.0        | Minimum headroom.  Used to compute a new threshold.                      |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
| max_headroom  | OVERCOVER_MAX_HEADROOM | --max-headroom (-M) | 0.0        | Maximum headroom.  Used to determine when the threshold must be updated. |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
|               | OVERCOVER_CONFIG       | --config (-c)       | *None*     | Specifies the name of the configuration file to use.                     |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
|               | OVERCOVER_READONLY     | --readonly (-r)     |            | Specifies that the configuration file should not be updated.             |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
|               | OVERCOVER_BUILD_ARG    | --build-arg (-b)    | *None*     | Specifies a build argument for package selection.                        |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
|               | OVERCOVER_SUMMARY      | --summary (-s)      |            | Specifies that per-package summary information should be emitted.        |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
|               | OVERCOVER_DETAILED     | --detailed (-d)     |            | Specifies that per-file coverage information should be emitted.          |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+
|               |                        | --help (-h)         |            | Emits help text describing how to use Overcover.                         |
+---------------+------------------------+---------------------+------------+--------------------------------------------------------------------------+

Key:

* *Required* - Option must be provided.
* *None* - No default, not required.