Documentation ¶
Overview ¶
Package hdrhistogram provides an implementation of Gil Tene's HDR Histogram data structure. The HDR Histogram allows for fast and accurate analysis of the extreme ranges of data with non-normal distributions, like latency.
Histograms are encoded using the HdrHistogram V2 format which is based on an adapted ZigZag LEB128 encoding where: consecutive zero counters are encoded as a negative number representing the count of consecutive zeros non zero counter values are encoded as a positive number A typical histogram (2 digits precision 1 usec to 1 day range) can be encoded in less than the typical MTU size of 1500 bytes.
The log format encodes into a single file, multiple histograms with optional shared meta data.
Index ¶
- Constants
- type Bar
- type Bracket
- type Histogram
- func (h *Histogram) ByteSize() int
- func (h *Histogram) CumulativeDistribution() []Bracket
- func (h *Histogram) CumulativeDistributionWithTicks(ticksPerHalfDistance int32) []Bracket
- func (h *Histogram) Distribution() (result []Bar)
- func (h *Histogram) Encode(version int32) (buffer []byte, err error)
- func (h *Histogram) EndTimeMs() int64
- func (h *Histogram) Equals(other *Histogram) bool
- func (h *Histogram) Export() *Snapshot
- func (h *Histogram) HighestTrackableValue() int64
- func (h *Histogram) LowestTrackableValue() int64
- func (h *Histogram) Max() int64
- func (h *Histogram) Mean() float64
- func (h *Histogram) Merge(from *Histogram) (dropped int64)
- func (h *Histogram) Min() int64
- func (h *Histogram) PercentilesPrint(writer io.Writer, ticksPerHalfDistance int32, valueScale float64) (outputWriter io.Writer, err error)
- func (h *Histogram) RecordCorrectedValue(v, expectedInterval int64) error
- func (h *Histogram) RecordValue(v int64) error
- func (h *Histogram) RecordValues(v, n int64) error
- func (h *Histogram) Reset()
- func (h *Histogram) SetEndTimeMs(endTimeMs int64)
- func (h *Histogram) SetStartTimeMs(startTimeMs int64)
- func (h *Histogram) SetTag(tag string)
- func (h *Histogram) SignificantFigures() int64
- func (h *Histogram) StartTimeMs() int64
- func (h *Histogram) StdDev() float64
- func (h *Histogram) Tag() string
- func (h *Histogram) TotalCount() int64
- func (h *Histogram) ValueAtPercentile(percentile float64) int64
- func (h *Histogram) ValueAtPercentiles(percentiles []float64) (values map[float64]int64)
- func (h *Histogram) ValueAtQuantile(q float64) int64
- func (h *Histogram) ValuesAreEquivalent(value1, value2 int64) (result bool)
- type HistogramLogOptions
- type HistogramLogReader
- func (hlr *HistogramLogReader) NextIntervalHistogram() (histogram *Histogram, err error)
- func (hlr *HistogramLogReader) NextIntervalHistogramWithRange(rangeStartTimeSec, rangeEndTimeSec float64, absolute bool) (histogram *Histogram, err error)
- func (hlr *HistogramLogReader) ObservedMax() bool
- func (hlr *HistogramLogReader) ObservedMin() bool
- func (hlr *HistogramLogReader) RangeObservedMax() int64
- func (hlr *HistogramLogReader) RangeObservedMin() int64
- type HistogramLogWriter
- func (lw *HistogramLogWriter) BaseTime() int64
- func (lw *HistogramLogWriter) OutputBaseTime(base_time_msec int64) (err error)
- func (lw *HistogramLogWriter) OutputComment(comment string) (err error)
- func (lw *HistogramLogWriter) OutputIntervalHistogram(histogram *Histogram) (err error)
- func (lw *HistogramLogWriter) OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions) (err error)
- func (lw *HistogramLogWriter) OutputLegend() (err error)
- func (lw *HistogramLogWriter) OutputLogFormatVersion() (err error)
- func (lw *HistogramLogWriter) OutputStartTime(start_time_msec int64) (err error)
- func (lw *HistogramLogWriter) SetBaseTime(baseTime int64)
- type Snapshot
- type WindowedHistogram
Examples ¶
Constants ¶
const ( V2EncodingCookieBase int32 = 0x1c849303 V2CompressedEncodingCookieBase int32 = 0x1c849304 ENCODING_HEADER_SIZE = 40 )
const HISTOGRAM_LOG_FORMAT_VERSION = "1.3"
const MsToNsRatio float64 = 1000000.0
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Histogram ¶
type Histogram struct {
// contains filtered or unexported fields
}
A Histogram is a lossy data structure used to record the distribution of non-normally distributed data (like latency) with a high degree of accuracy and a bounded degree of precision.
func Decode ¶ added in v1.0.0
Decode returns a new Histogram by decoding it from a String containing a base64 encoded compressed histogram representation.
func Import ¶
Import returns a new Histogram populated from the Snapshot data (which the caller must stop accessing).
func New ¶
func New(lowestDiscernibleValue, highestTrackableValue int64, numberOfSignificantValueDigits int) *Histogram
Construct a Histogram given the Lowest and Highest values to be tracked and a number of significant decimal digits.
Providing a lowestDiscernibleValue is useful in situations where the units used for the histogram's values are much smaller that the minimal accuracy required. E.g. when tracking time values stated in nanosecond units, where the minimal accuracy required is a microsecond, the proper value for lowestDiscernibleValue would be 1000.
Note: the numberOfSignificantValueDigits must be [1,5]. If lower than 1 the numberOfSignificantValueDigits will be forced to 1, and if higher than 5 the numberOfSignificantValueDigits will be forced to 5.
Example ¶
This latency Histogram could be used to track and analyze the counts of observed integer values between 1 us and 30000000 us ( 30 secs ) while maintaining a value precision of 4 significant digits across that range, translating to a value resolution of :
- 1 microsecond up to 10 milliseconds,
- 100 microsecond (or better) from 10 milliseconds up to 10 seconds,
- 300 microsecond (or better) from 10 seconds up to 30 seconds,
nolint
lH := hdrhistogram.New(1, 30000000, 4) input := []int64{ 459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317, 3964974, 12718782, } for _, sample := range input { lH.RecordValue(sample) } fmt.Printf("Percentile 50: %d\n", lH.ValueAtQuantile(50.0))
Output: Percentile 50: 931423
func (*Histogram) ByteSize ¶
ByteSize returns an estimate of the amount of memory allocated to the histogram in bytes.
N.B.: This does not take into account the overhead for slices, which are small, constant, and specific to the compiler version.
func (*Histogram) CumulativeDistribution ¶
CumulativeDistribution returns an ordered list of brackets of the distribution of recorded values.
func (*Histogram) CumulativeDistributionWithTicks ¶ added in v1.0.0
CumulativeDistribution returns an ordered list of brackets of the distribution of recorded values.
func (*Histogram) Distribution ¶
Distribution returns an ordered list of bars of the distribution of recorded values, counts can be normalized to a probability
func (*Histogram) Encode ¶ added in v1.0.0
Encode returns a snapshot view of the Histogram. The snapshot is compact binary representations of the state of the histogram. They are intended to be used for archival or transmission to other systems for further analysis.
func (*Histogram) Export ¶
Export returns a snapshot view of the Histogram. This can be later passed to Import to construct a new Histogram with the same state.
func (*Histogram) HighestTrackableValue ¶
HighestTrackableValue returns the upper bound on values that will be added to the histogram
func (*Histogram) LowestTrackableValue ¶
LowestTrackableValue returns the lower bound on values that will be added to the histogram
func (*Histogram) Merge ¶
Merge merges the data stored in the given histogram with the receiver, returning the number of recorded values which had to be dropped.
func (*Histogram) PercentilesPrint ¶ added in v1.0.0
func (h *Histogram) PercentilesPrint(writer io.Writer, ticksPerHalfDistance int32, valueScale float64) (outputWriter io.Writer, err error)
Output the percentiles distribution in a text format
Example ¶
The following example details the creation of an histogram used to track and analyze the counts of observed integer values between 0 us and 30000000 us ( 30 secs ) and the printing of the percentile output format nolint
lH := hdrhistogram.New(1, 30000000, 3) input := []int64{ 459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317, 3964974, 12718782, } for _, sample := range input { lH.RecordValue(sample) } lH.PercentilesPrint(os.Stdout, 1, 1.0)
Output: Value Percentile TotalCount 1/(1-Percentile) 460031.000 0.000000 1 1.00 931839.000 0.500000 5 2.00 2478079.000 0.750000 8 4.00 3966975.000 0.875000 9 8.00 12722175.000 0.937500 10 16.00 12722175.000 1.000000 10 inf #[Mean = 2491481.600, StdDeviation = 3557920.109] #[Max = 12722175.000, Total count = 10] #[Buckets = 15, SubBuckets = 2048]
func (*Histogram) RecordCorrectedValue ¶
RecordCorrectedValue records the given value, correcting for stalls in the recording process. This only works for processes which are recording values at an expected interval (e.g., doing jitter analysis). Processes which are recording ad-hoc values (e.g., latency for incoming requests) can't take advantage of this.
func (*Histogram) RecordValue ¶
RecordValue records the given value, returning an error if the value is out of range.
Example ¶
This latency Histogram could be used to track and analyze the counts of observed integer values between 0 us and 30000000 us ( 30 secs ) while maintaining a value precision of 3 significant digits across that range, translating to a value resolution of :
- 1 microsecond up to 1 millisecond,
- 1 millisecond (or better) up to one second,
- 1 second (or better) up to it's maximum tracked value ( 30 seconds ).
nolint
lH := hdrhistogram.New(1, 30000000, 3) input := []int64{ 459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317, 3964974, 12718782, } for _, sample := range input { lH.RecordValue(sample) } fmt.Printf("Percentile 50: %d\n", lH.ValueAtQuantile(50.0))
Output: Percentile 50: 931839
func (*Histogram) RecordValues ¶
RecordValues records n occurrences of the given value, returning an error if the value is out of range.
func (*Histogram) Reset ¶
func (h *Histogram) Reset()
Reset deletes all recorded values and restores the histogram to its original state.
func (*Histogram) SetEndTimeMs ¶ added in v1.0.0
func (*Histogram) SetStartTimeMs ¶ added in v1.0.0
func (*Histogram) SignificantFigures ¶
SignificantFigures returns the significant figures used to create the histogram
func (*Histogram) StartTimeMs ¶ added in v1.0.0
func (*Histogram) StdDev ¶
StdDev returns the approximate standard deviation of the recorded values.
func (*Histogram) TotalCount ¶
TotalCount returns total number of values recorded.
func (*Histogram) ValueAtPercentile ¶ added in v1.1.0
ValueAtPercentile returns the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.
The passed percentile must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.
Returns 0 if no recorded values exist.
func (*Histogram) ValueAtPercentiles ¶ added in v1.1.0
ValueAtPercentiles, given an slice of percentiles returns a map containing for each passed percentile, the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.
Each element in the given an slice of percentiles must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.
Returns a map of 0's if no recorded values exist.
Example ¶
When doing an percentile analysis we normally require more than one percentile to be calculated for the given histogram.
When that is the case ValueAtPercentiles() will deeply optimize the total time to retrieve the percentiles vs the other option which is multiple calls to ValueAtQuantile().
nolint
histogram := hdrhistogram.New(1, 30000000, 3) for i := 0; i < 1000000; i++ { histogram.RecordValue(int64(i)) } percentileValuesMap := histogram.ValueAtPercentiles([]float64{50.0, 95.0, 99.0, 99.9}) fmt.Printf("Percentile 50: %d\n", percentileValuesMap[50.0]) fmt.Printf("Percentile 95: %d\n", percentileValuesMap[95.0]) fmt.Printf("Percentile 99: %d\n", percentileValuesMap[99.0]) fmt.Printf("Percentile 99.9: %d\n", percentileValuesMap[99.9])
Output: Percentile 50: 500223 Percentile 95: 950271 Percentile 99: 990207 Percentile 99.9: 999423
func (*Histogram) ValueAtQuantile ¶
ValueAtQuantile returns the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.
The passed quantile must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.
Returns 0 if no recorded values exist.
func (*Histogram) ValuesAreEquivalent ¶ added in v1.0.1
Determine if two values are equivalent with the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
type HistogramLogOptions ¶ added in v1.0.0
type HistogramLogOptions struct {
// contains filtered or unexported fields
}
func DefaultHistogramLogOptions ¶ added in v1.0.0
func DefaultHistogramLogOptions() *HistogramLogOptions
type HistogramLogReader ¶ added in v1.0.0
type HistogramLogReader struct {
// contains filtered or unexported fields
}
func NewHistogramLogReader ¶ added in v1.0.0
func NewHistogramLogReader(log io.Reader) *HistogramLogReader
Example ¶
The log format encodes into a single file, multiple histograms with optional shared meta data. The following example showcases reading a log file into a slice of histograms nolint
package main import ( "bytes" "fmt" hdrhistogram "github.com/HdrHistogram/hdrhistogram-go" "io/ioutil" ) func main() { dat, _ := ioutil.ReadFile("./test/tagged-Log.logV2.hlog") r := bytes.NewReader(dat) // Create a histogram log reader reader := hdrhistogram.NewHistogramLogReader(r) var histograms []*hdrhistogram.Histogram = make([]*hdrhistogram.Histogram, 0) // Read all histograms in the file for hist, err := reader.NextIntervalHistogram(); hist != nil && err == nil; hist, err = reader.NextIntervalHistogram() { histograms = append(histograms, hist) } fmt.Printf("Read a total of %d histograms\n", len(histograms)) min := reader.RangeObservedMin() max := reader.RangeObservedMax() sigdigits := 3 overallHistogram := hdrhistogram.New(min, max, sigdigits) //// We can then merge all histograms into one and retrieve overall metrics for _, hist := range histograms { overallHistogram.Merge(hist) } fmt.Printf("Overall count: %d samples\n", overallHistogram.TotalCount()) fmt.Printf("Overall Percentile 50: %d\n", overallHistogram.ValueAtQuantile(50.0)) }
Output: Read a total of 42 histograms Overall count: 32290 samples Overall Percentile 50: 344319
func (*HistogramLogReader) NextIntervalHistogram ¶ added in v1.0.0
func (hlr *HistogramLogReader) NextIntervalHistogram() (histogram *Histogram, err error)
func (*HistogramLogReader) NextIntervalHistogramWithRange ¶ added in v1.0.0
func (hlr *HistogramLogReader) NextIntervalHistogramWithRange(rangeStartTimeSec, rangeEndTimeSec float64, absolute bool) (histogram *Histogram, err error)
func (*HistogramLogReader) ObservedMax ¶ added in v1.0.0
func (hlr *HistogramLogReader) ObservedMax() bool
func (*HistogramLogReader) ObservedMin ¶ added in v1.0.0
func (hlr *HistogramLogReader) ObservedMin() bool
func (*HistogramLogReader) RangeObservedMax ¶ added in v1.0.0
func (hlr *HistogramLogReader) RangeObservedMax() int64
Returns the overall observed max limit ( up to the current point ) of the read histograms
func (*HistogramLogReader) RangeObservedMin ¶ added in v1.0.0
func (hlr *HistogramLogReader) RangeObservedMin() int64
Returns the overall observed min limit ( up to the current point ) of the read histograms
type HistogramLogWriter ¶ added in v1.0.0
type HistogramLogWriter struct {
// contains filtered or unexported fields
}
func NewHistogramLogWriter ¶ added in v1.0.0
func NewHistogramLogWriter(log io.Writer) *HistogramLogWriter
Example ¶
The log format encodes into a single file, multiple histograms with optional shared meta data. The following example showcases writing multiple histograms into a log file and then processing them again to confirm a proper encode-decode flow nolint
package main import ( "bytes" "fmt" hdrhistogram "github.com/HdrHistogram/hdrhistogram-go" "io/ioutil" ) func main() { var buff bytes.Buffer // Create a histogram log writer to write to a bytes.Buffer writer := hdrhistogram.NewHistogramLogWriter(&buff) writer.OutputLogFormatVersion() writer.OutputStartTime(0) writer.OutputLegend() // Lets create 3 distinct histograms to exemply the logwriter features // each one with a time-frame of 60 secs ( 60000 ms ) hist1 := hdrhistogram.New(1, 30000000, 3) hist1.SetStartTimeMs(0) hist1.SetEndTimeMs(60000) for _, sample := range []int64{10, 20, 30, 40} { hist1.RecordValue(sample) } hist2 := hdrhistogram.New(1, 3000, 3) hist1.SetStartTimeMs(60001) hist1.SetEndTimeMs(120000) for _, sample := range []int64{50, 70, 80, 60} { hist2.RecordValue(sample) } hist3 := hdrhistogram.New(1, 30000, 3) hist1.SetStartTimeMs(120001) hist1.SetEndTimeMs(180000) for _, sample := range []int64{90, 100} { hist3.RecordValue(sample) } writer.OutputIntervalHistogram(hist1) writer.OutputIntervalHistogram(hist2) writer.OutputIntervalHistogram(hist3) ioutil.WriteFile("example.logV2.hlog", buff.Bytes(), 0644) // read check // Lets read all again and confirm that the total sample count is 10 dat, _ := ioutil.ReadFile("example.logV2.hlog") r := bytes.NewReader(dat) // Create a histogram log reader reader := hdrhistogram.NewHistogramLogReader(r) var histograms []*hdrhistogram.Histogram = make([]*hdrhistogram.Histogram, 0) // Read all histograms in the file for hist, err := reader.NextIntervalHistogram(); hist != nil && err == nil; hist, err = reader.NextIntervalHistogram() { histograms = append(histograms, hist) } fmt.Printf("Read a total of %d histograms\n", len(histograms)) min := reader.RangeObservedMin() max := reader.RangeObservedMax() sigdigits := 3 overallHistogram := hdrhistogram.New(min, max, sigdigits) //// We can then merge all histograms into one and retrieve overall metrics for _, hist := range histograms { overallHistogram.Merge(hist) } fmt.Printf("Overall count: %d samples\n", overallHistogram.TotalCount()) }
Output: Read a total of 3 histograms Overall count: 10 samples
func (*HistogramLogWriter) BaseTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) BaseTime() int64
Return the current base time offset
func (*HistogramLogWriter) OutputBaseTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputBaseTime(base_time_msec int64) (err error)
Log a base time in the log. Base time is represented as seconds since epoch with up to 3 decimal places. Line starts with the leading text '#[BaseTime:'
func (*HistogramLogWriter) OutputComment ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputComment(comment string) (err error)
Log a comment to the log. A comment is any line that leads with '#' that is not matched by the BaseTime or StartTime formats. Comments are ignored when parsed.
func (*HistogramLogWriter) OutputIntervalHistogram ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputIntervalHistogram(histogram *Histogram) (err error)
Output an interval histogram, using the start/end timestamp indicated in the histogram, and the [optional] tag associated with the histogram. The histogram start and end timestamps are assumed to be in msec units
By convention, histogram start/end time are generally stamped with absolute times in msec since the epoch. For logging with absolute time stamps, the base time would remain zero ( default ). For logging with relative time stamps (time since a start point), the base time should be set with SetBaseTime(baseTime int64)
The max value in the histogram will be reported scaled down by a default maxValueUnitRatio of 1000000.0 (which is the msec : nsec ratio). If you need to specify a different start/end timestamp or a different maxValueUnitRatio you should use OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions)
func (*HistogramLogWriter) OutputIntervalHistogramWithLogOptions ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions) (err error)
Output an interval histogram, with the given timestamp information and the [optional] tag associated with the histogram
If you specify non-nil logOptions, and non-zero start timestamp, the the specified timestamp information will be used, and the start timestamp information in the actual histogram will be ignored. If you specify non-nil logOptions, and non-zero start timestamp, the the specified timestamp information will be used, and the end timestamp information in the actual histogram will be ignored. If you specify non-nil logOptions, The max value reported with the interval line will be scaled by the given maxValueUnitRatio, otherwise a default maxValueUnitRatio of 1,000,000 (which is the msec : nsec ratio) will be used.
By convention, histogram start/end time are generally stamped with absolute times in msec since the epoch. For logging with absolute time stamps, the base time would remain zero ( default ). For logging with relative time stamps (time since a start point), the base time should be set with SetBaseTime(baseTime int64)
func (*HistogramLogWriter) OutputLegend ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputLegend() (err error)
Output a legend line to the log. Human readable column headers. Ignored when parsed.
func (*HistogramLogWriter) OutputLogFormatVersion ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputLogFormatVersion() (err error)
Output a log format version to the log.
func (*HistogramLogWriter) OutputStartTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputStartTime(start_time_msec int64) (err error)
Log a start time in the log. Start time is represented as seconds since epoch with up to 3 decimal places. Line starts with the leading text '#[StartTime:'
func (*HistogramLogWriter) SetBaseTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) SetBaseTime(baseTime int64)
Set a base time to subtract from supplied histogram start/end timestamps when logging based on histogram timestamps. baseTime is expected to be in msec since the epoch, as histogram start/end times are typically stamped with absolute times in msec since the epoch.
type Snapshot ¶
type Snapshot struct { LowestTrackableValue int64 HighestTrackableValue int64 SignificantFigures int64 Counts []int64 }
A Snapshot is an exported view of a Histogram, useful for serializing them. A Histogram can be constructed from it by passing it to Import.
type WindowedHistogram ¶
type WindowedHistogram struct { Current *Histogram // contains filtered or unexported fields }
A WindowedHistogram combines histograms to provide windowed statistics.
func NewWindowed ¶
func NewWindowed(n int, minValue, maxValue int64, sigfigs int) *WindowedHistogram
NewWindowed creates a new WindowedHistogram with N underlying histograms with the given parameters.
func (*WindowedHistogram) Merge ¶
func (w *WindowedHistogram) Merge() *Histogram
Merge returns a histogram which includes the recorded values from all the sections of the window.
func (*WindowedHistogram) Rotate ¶
func (w *WindowedHistogram) Rotate()
Rotate resets the oldest histogram and rotates it to be used as the current histogram.