metrics

README

package metrics

package metrics provides a set of uniform interfaces for service instrumentation. It has counters, gauges, and histograms, and provides adapters to popular metrics packages, like expvar, StatsD, and Prometheus.

Rationale

Code instrumentation is absolutely essential to achieve observability into a distributed system. Metrics and instrumentation tools have coalesced around a few well-defined idioms. package metrics provides a common, minimal interface those idioms for service authors.

Usage

A simple counter, exported via expvar.

import (
	"github.com/inturn/kit/metrics"
	"github.com/inturn/kit/metrics/expvar"
)

func main() {
	var myCount metrics.Counter
	myCount = expvar.NewCounter("my_count")
	myCount.Add(1)
}

A histogram for request duration, exported via a Prometheus summary with dynamically-computed quantiles.

import (
	"time"

	stdprometheus "github.com/prometheus/client_golang/prometheus"

	"github.com/inturn/kit/metrics"
	"github.com/inturn/kit/metrics/prometheus"
)

func main() {
	var dur metrics.Histogram = prometheus.NewSummaryFrom(stdprometheus.SummaryOpts{
		Namespace: "myservice",
		Subsystem: "api",
		Name:     "request_duration_seconds",
		Help:     "Total time spent serving requests.",
	}, []string{})
	// ...
}

func handleRequest(dur metrics.Histogram) {
	defer func(begin time.Time) { dur.Observe(time.Since(begin).Seconds()) }(time.Now())
	// handle request
}

A gauge for the number of goroutines currently running, exported via StatsD.

import (
	"net"
	"os"
	"runtime"
	"time"

	"github.com/inturn/kit/metrics"
	"github.com/inturn/kit/metrics/statsd"
)

func main() {
	statsd := statsd.New("foo_svc.", log.NewNopLogger())
	report := time.NewTicker(5 * time.Second)
	defer report.Stop()
	go statsd.SendLoop(report.C, "tcp", "statsd.internal:8125")
	goroutines := statsd.NewGauge("goroutine_count")
	go exportGoroutines(goroutines)
	// ...
}

func exportGoroutines(g metrics.Gauge) {
	for range time.Tick(time.Second) {
		g.Set(float64(runtime.NumGoroutine()))
	}
}

For more information, see the package documentation.

Documentation

Overview

Package metrics provides a framework for application instrumentation. It's primarily designed to help you get started with good and robust instrumentation, and to help you migrate from a less-capable system like Graphite to a more-capable system like Prometheus. If your organization has already standardized on an instrumentation system like Prometheus, and has no plans to change, it may make sense to use that system's instrumentation library directly.

This package provides three core metric abstractions (Counter, Gauge, and Histogram) and implementations for almost all common instrumentation backends. Each metric has an observation method (Add, Set, or Observe, respectively) used to record values, and a With method to "scope" the observation by various parameters. For example, you might have a Histogram to record request durations, parameterized by the method that's being called.

var requestDuration metrics.Histogram
// ...
requestDuration.With("method", "MyMethod").Observe(time.Since(begin))

This allows a single high-level metrics object (requestDuration) to work with many code paths somewhat dynamically. The concept of With is fully supported in some backends like Prometheus, and not supported in other backends like Graphite. So, With may be a no-op, depending on the concrete implementation you choose. Please check the implementation to know for sure. For implementations that don't provide With, it's necessary to fully parameterize each metric in the metric name, e.g.

// Statsd
c := statsd.NewCounter("request_duration_MyMethod_200")
c.Add(1)

// Prometheus
c := prometheus.NewCounter(stdprometheus.CounterOpts{
    Name: "request_duration",
    ...
}, []string{"method", "status_code"})
c.With("method", "MyMethod", "status_code", strconv.Itoa(code)).Add(1)

Usage

Metrics are dependencies, and should be passed to the components that need them in the same way you'd construct and pass a database handle, or reference to another component. Metrics should *not* be created in the global scope. Instead, instantiate metrics in your func main, using whichever concrete implementation is appropriate for your organization.

latency := prometheus.NewSummaryFrom(stdprometheus.SummaryOpts{
    Namespace: "myteam",
    Subsystem: "foosvc",
    Name:      "request_latency_seconds",
    Help:      "Incoming request latency in seconds.",
}, []string{"method", "status_code"})

Write your components to take the metrics they will use as parameters to their constructors. Use the interface types, not the concrete types. That is,

// NewAPI takes metrics.Histogram, not *prometheus.Summary
func NewAPI(s Store, logger log.Logger, latency metrics.Histogram) *API {
    // ...
}

func (a *API) ServeFoo(w http.ResponseWriter, r *http.Request) {
    begin := time.Now()
    // ...
    a.latency.Observe(time.Since(begin).Seconds())
}

Finally, pass the metrics as dependencies when building your object graph. This should happen in func main, not in the global scope.

api := NewAPI(store, logger, latency)
http.ListenAndServe("/", api)

Note that metrics are "write-only" interfaces.

Implementation details

All metrics are safe for concurrent use. Considerable design influence has been taken from https://github.com/codahale/metrics and https://prometheus.io.

Each telemetry system has different semantics for label values, push vs. pull, support for histograms, etc. These properties influence the design of their respective packages. This table attempts to summarize the key points of distinction.

SYSTEM      DIM  COUNTERS               GAUGES                 HISTOGRAMS
dogstatsd   n    batch, push-aggregate  batch, push-aggregate  native, batch, push-each
statsd      1    batch, push-aggregate  batch, push-aggregate  native, batch, push-each
graphite    1    batch, push-aggregate  batch, push-aggregate  synthetic, batch, push-aggregate
expvar      1    atomic                 atomic                 synthetic, batch, in-place expose
influx      n    custom                 custom                 custom
prometheus  n    native                 native                 native
pcp         1    native                 native                 native
cloudwatch  n    batch push-aggregate   batch push-aggregate   synthetic, batch, push-aggregate

Index

• type Counter
• type Gauge
• type Histogram
• type Timer
  • func NewTimer(h Histogram) *Timer
  • func (t *Timer) ObserveDuration()
  • func (t *Timer) Unit(u time.Duration)

Types

type Counter

type Counter interface {
	With(labelValues ...string) Counter
	Add(delta float64)
}

Counter describes a metric that accumulates values monotonically. An example of a counter is the number of received HTTP requests.

type Gauge

type Gauge interface {
	With(labelValues ...string) Gauge
	Set(value float64)
	Add(delta float64)
}

Gauge describes a metric that takes specific values over time. An example of a gauge is the current depth of a job queue.

type Histogram

type Histogram interface {
	With(labelValues ...string) Histogram
	Observe(value float64)
}

Histogram describes a metric that takes repeated observations of the same kind of thing, and produces a statistical summary of those observations, typically expressed as quantiles or buckets. An example of a histogram is HTTP request latencies.

type Timer

type Timer struct {
	// contains filtered or unexported fields
}

Timer acts as a stopwatch, sending observations to a wrapped histogram. It's a bit of helpful syntax sugar for h.Observe(time.Since(x)).

func NewTimer

func NewTimer(h Histogram) *Timer

NewTimer wraps the given histogram and records the current time.

func (*Timer) ObserveDuration

func (t *Timer) ObserveDuration()

ObserveDuration captures the number of seconds since the timer was constructed, and forwards that observation to the histogram.

func (*Timer) Unit

func (t *Timer) Unit(u time.Duration)

Unit sets the unit of the float64 emitted by the timer. By default, the timer emits seconds.