README ¶
loghisto
A metric system for high performance counters and histograms. Unlike popular metric systems today, this does not destroy the accuracy of histograms by sampling. Instead, a logarithmic bucketing function compresses values, generally within 1% of their true value (although between 0 and 1 the precision loss may not be within this boundary). This allows for extreme compression, which allows us to calculate arbitrarily high percentiles with no loss of accuracy - just a small amount of precision. This is particularly useful for highly-clustered events that are tolerant of a small precision loss, but for which you REALLY care about what the tail looks like, such as measuring latency across a distributed system.
Copied out of my work for the CockroachDB metrics system. Based on an algorithm created by Keith Frost.
running a print benchmark for quick analysis
package main
import (
"runtime"
"github.com/spacejam/loghisto"
)
func benchmark() {
// do some stuff
}
func main() {
numCPU := runtime.NumCPU()
runtime.GOMAXPROCS(numCPU)
desiredConcurrency := uint(100)
loghisto.PrintBenchmark("benchmark1234", desiredConcurrency, benchmark)
}
results in something like this printed to stdout each second:
2014-12-11 21:41:45 -0500 EST
benchmark1234_count: 2.0171025e+07
benchmark1234_max: 2.4642914167480484e+07
benchmark1234_99.99: 4913.768840299134
benchmark1234_99.9: 1001.2472422902518
benchmark1234_99: 71.24044000732538
benchmark1234_95: 67.03348428941965
benchmark1234_90: 65.68633104092515
benchmark1234_75: 63.07152259993664
benchmark1234_50: 58.739891704145194
benchmark1234_min: -657.5233632152207 // Corollary: time.Since(time.Now()) is often < 0
benchmark1234_sum: 1.648051169322668e+09
benchmark1234_avg: 81.70388809307748
benchmark1234_agg_avg: 89
benchmark1234_agg_count: 6.0962226e+07
benchmark1234_agg_sum: 5.454779078e+09
sys.Alloc: 1.132672e+06
sys.NumGC: 5741
sys.PauseTotalNs: 1.569390954e+09
sys.NumGoroutine: 113
adding an embedded metric system to your code
import (
"time"
"fmt"
"github.com/spacejam/loghisto"
)
func ExampleMetricSystem() {
// Create metric system that reports once a minute, and includes stats
// about goroutines, memory usage and GC.
includeGoProcessStats := true
ms := loghisto.NewMetricSystem(time.Minute, includeGoProcessStats)
ms.Start()
// create a channel that subscribes to metrics as they are produced once
// per minute.
// NOTE: if you allow this channel to fill up, the metric system will NOT
// block, and will FORGET about your channel if you fail to unblock the
// channel after 3 configured intervals (in this case 3 minutes) rather
// than causing a memory leak.
myMetricStream := make(chan *loghisto.ProcessedMetricSet, 2)
ms.SubscribeToProcessedMetrics(myMetricStream)
// create some metrics
timeToken := ms.StartTimer("time for creating a counter and histo")
ms.Counter("some event", 1)
ms.Histogram("some measured thing", 123)
timeToken.Stop()
for m := range myMetricStream {
fmt.Printf("number of goroutines: %f\n", m.Metrics["sys.NumGoroutine"])
}
// if you want to manually unsubscribe from the metric stream
ms.UnsubscribeFromProcessedMetrics(myMetricStream)
// to stop and clean up your metric system
ms.Stop()
}
automatically sending your metrics to OpenTSDB, KairosDB or Graphite
func ExampleExternalSubmitter() {
includeGoProcessStats := true
ms := NewMetricSystem(time.Minute, includeGoProcessStats)
ms.Start()
// graphite
s := NewSubmitter(ms, GraphiteProtocol, "tcp", "localhost:7777")
s.Start()
// opentsdb / kairosdb
s := NewSubmitter(ms, OpenTSDBProtocol, "tcp", "localhost:7777")
s.Start()
// to tear down:
s.Shutdown()
}
See code for the Graphite/OpenTSDB protocols for adding your own output plugins, it's pretty simple.
Documentation ¶
Index ¶
- Variables
- func GraphiteProtocol(ms *ProcessedMetricSet) []byte
- func OpenTSDBProtocol(ms *ProcessedMetricSet) []byte
- func PrintBenchmark(name string, concurrency uint, op func())
- type MetricSystem
- func (ms *MetricSystem) Counter(name string, amount uint64)
- func (ms *MetricSystem) DeregisterGaugeFunc(name string)
- func (ms *MetricSystem) Histogram(name string, value float64)
- func (ms *MetricSystem) RegisterGaugeFunc(name string, f func() float64)
- func (ms *MetricSystem) SpecifyPercentiles(percentiles map[string]float64)
- func (ms *MetricSystem) Start()
- func (ms *MetricSystem) StartTimer(name string) TimerToken
- func (ms *MetricSystem) Stop()
- func (ms *MetricSystem) SubscribeToProcessedMetrics(metricStream chan *ProcessedMetricSet)
- func (ms *MetricSystem) SubscribeToRawMetrics(metricStream chan *RawMetricSet)
- func (ms *MetricSystem) UnsubscribeFromProcessedMetrics(metricStream chan *ProcessedMetricSet)
- func (ms *MetricSystem) UnsubscribeFromRawMetrics(metricStream chan *RawMetricSet)
- type ProcessedMetricSet
- type RawMetricSet
- type Submitter
- type TimerToken
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var Metrics = NewMetricSystem(60*time.Second, true)
Metrics is the default metric system, which collects and broadcasts metrics to subscribers once every 60 seconds. Also includes default system stats.
Functions ¶
func GraphiteProtocol ¶
func GraphiteProtocol(ms *ProcessedMetricSet) []byte
GraphiteProtocol generates a wire representation of a ProcessedMetricSet for submission to a Graphite Carbon instance using the plaintext protocol.
func OpenTSDBProtocol ¶
func OpenTSDBProtocol(ms *ProcessedMetricSet) []byte
OpenTSDBProtocol generates a wire representation of a ProcessedMetricSet for submission to an OpenTSDB instance.
func PrintBenchmark ¶
PrintBenchmark will run the provided function at the specified concurrency, time the operation, and once per second write the following information to standard out:
2014-08-09 17:44:57 -0400 EDT raft_AppendLogEntries_count: 16488 raft_AppendLogEntries_max: 3.982478339757623e+07 raft_AppendLogEntries_99.99: 3.864778314316012e+07 raft_AppendLogEntries_99.9: 3.4366224772310276e+06 raft_AppendLogEntries_99: 2.0228126576114902e+06 raft_AppendLogEntries_50: 469769.7083161708 raft_AppendLogEntries_min: 129313.15075081984 raft_AppendLogEntries_sum: 9.975892639594093e+09 raft_AppendLogEntries_avg: 605039.5827022133 raft_AppendLogEntries_agg_avg: 618937 raft_AppendLogEntries_agg_count: 121095 raft_AppendLogEntries_agg_sum: 7.4950269894e+10 sys.Alloc: 997328 sys.NumGC: 1115 sys.PauseTotalNs: 2.94946542e+08 sys.NumGoroutine: 26
Types ¶
type MetricSystem ¶
type MetricSystem struct {
// contains filtered or unexported fields
}
MetricSystem facilitates the collection and distribution of metrics.
Example ¶
Output: total range splits during the process lifetime present range splits in this period present some_ipc 99.9th percentile present some_ipc max present some_ipc calls this period present some_ipc calls during the process lifetime present some_ipc total latency this period present some_ipc mean this period present some_ipc aggregate man present time spent submitting metrics this period present number of goroutines present time spent in GC present
func NewMetricSystem ¶
func NewMetricSystem(interval time.Duration, sysStats bool) *MetricSystem
NewMetricSystem returns a new metric system that collects and broadcasts metrics after each interval.
func (*MetricSystem) Counter ¶
func (ms *MetricSystem) Counter(name string, amount uint64)
Counter is used for recording a running count of the total occurrences of a particular event. A rate is also exported for the amount that a counter has increased during an interval of this MetricSystem.
func (*MetricSystem) DeregisterGaugeFunc ¶
func (ms *MetricSystem) DeregisterGaugeFunc(name string)
DeregisterGaugeFunc deregisters a function for the <name> metric.
func (*MetricSystem) Histogram ¶
func (ms *MetricSystem) Histogram(name string, value float64)
Histogram is used for generating rich metrics, such as percentiles, from periodically occurring continuous values.
func (*MetricSystem) RegisterGaugeFunc ¶
func (ms *MetricSystem) RegisterGaugeFunc(name string, f func() float64)
RegisterGaugeFunc registers a function to be called at each interval whose return value will be used to populate the <name> metric.
func (*MetricSystem) SpecifyPercentiles ¶
func (ms *MetricSystem) SpecifyPercentiles(percentiles map[string]float64)
SpecifyPercentiles allows users to override the default collected and reported percentiles.
func (*MetricSystem) Start ¶
func (ms *MetricSystem) Start()
Start spawns a goroutine for merging metrics into caches from metric submitters, and a reaper goroutine that harvests metrics at the default interval of every 60 seconds.
func (*MetricSystem) StartTimer ¶
func (ms *MetricSystem) StartTimer(name string) TimerToken
StartTimer begins a timer and returns a token which is required for halting the timer. This allows for concurrent timings under the same name.
func (*MetricSystem) SubscribeToProcessedMetrics ¶
func (ms *MetricSystem) SubscribeToProcessedMetrics( metricStream chan *ProcessedMetricSet)
SubscribeToProcessedMetrics registers a channel to receive ProcessedMetricSets periodically generated by reaper at each interval.
func (*MetricSystem) SubscribeToRawMetrics ¶
func (ms *MetricSystem) SubscribeToRawMetrics(metricStream chan *RawMetricSet)
SubscribeToRawMetrics registers a channel to receive RawMetricSets periodically generated by reaper at each interval.
func (*MetricSystem) UnsubscribeFromProcessedMetrics ¶
func (ms *MetricSystem) UnsubscribeFromProcessedMetrics( metricStream chan *ProcessedMetricSet)
UnsubscribeFromProcessedMetrics registers a channel to receive ProcessedMetricSets periodically generated by reaper at each interval.
func (*MetricSystem) UnsubscribeFromRawMetrics ¶
func (ms *MetricSystem) UnsubscribeFromRawMetrics( metricStream chan *RawMetricSet)
UnsubscribeFromRawMetrics registers a channel to receive RawMetricSets periodically generated by reaper at each interval.
type ProcessedMetricSet ¶
ProcessedMetricSet contains human-readable metrics that may also be suitable for storage in time-series databases.
type RawMetricSet ¶
type RawMetricSet struct { Time time.Time Counters map[string]uint64 Rates map[string]uint64 Histograms map[string]map[int16]*uint64 Gauges map[string]float64 }
RawMetricSet contains metrics in a form that supports generation of percentiles and other rich statistics.
type Submitter ¶
type Submitter struct { DestinationNetwork string DestinationAddress string // contains filtered or unexported fields }
Submitter encapsulates the state of a metric submitter.
func NewSubmitter ¶
func NewSubmitter(metricSystem *MetricSystem, serializer func(*ProcessedMetricSet) []byte, destinationNetwork string, destinationAddress string) *Submitter
NewSubmitter creates a Submitter that receives metrics off of a specified metric channel, serializes them using the provided serialization function, and attempts to send them to the specified destination.
type TimerToken ¶
type TimerToken struct { Name string Start time.Time MetricSystem *MetricSystem }
TimerToken facilitates concurrent timings of durations of the same label.
func (*TimerToken) Stop ¶
func (tt *TimerToken) Stop() time.Duration
Stop stops a timer given by StartTimer, submits a Histogram of its duration in nanoseconds, and returns its duration in nanoseconds.