Documentation
¶
Overview ¶
Package events contains a number of different data types and formats that you can use to populate ftdc metrics series.
Custom and CustomPoint ¶
The "custom" types allow you to construct arbirary key-value pairs without using maps and have them be well represented in FTDC output. Populate and interact with the data sequence as a slice of key (string) value (numbers) pairs, which are marshaled in the database as an object as a mapping of strings to numbers. The type provides some additional helpers for manipulating these data.
Histogram ¶
The histogram representation is broadly similar to the Performance structure but stores data in a histogram format, which offers a high fidelity representation of a very large number of raw events without the storage overhead. In general, use histograms to collect data for operation with throughput in the thousands or more operations per second.
Performance Points ¶
The Performance type represents a unified event to track an operation in a performance test. These events record three types of metrics: counters, timers, and gauges. Counters record the number of operations in different ways, including test iterations, logical operation counts, operation size (bytes), and error rate. Timers include both the latency of the core operation, for use in calculating latencies as well as the total taken which may be useful in calculating throughput. Finally gauges, capture changes in state or other information about the environment including the number of threads used in the test, or a failed Boolean when a test is aware of its own failure.
Recorder ¶
The Recorder interface provides an interface for workloads and operations to to collect data about their internal state, without requiring workloads to be concerned with data retenation, storage, or compression. The implementations of Recorder have different strategies for data collection and persistence so that tests can easily change data collection strategies without modifying the test.
Index ¶
- type Collector
- func NewBasicCollector(fc ftdc.Collector) Collector
- func NewIntervalCollector(fc ftdc.Collector, interval time.Duration) Collector
- func NewPassthroughCollector(fc ftdc.Collector) Collector
- func NewRandomSamplingCollector(fc ftdc.Collector, sumAll bool, percent int) Collector
- func NewSamplingCollector(fc ftdc.Collector, n int) Collector
- func NewSynchronizedCollector(coll Collector) Collector
- type Custom
- func (ps *Custom) Add(key string, value interface{}) error
- func (ps Custom) Len() int
- func (ps Custom) Less(i, j int) bool
- func (ps Custom) MarshalBSON() ([]byte, error)
- func (ps Custom) MarshalDocument() (*birch.Document, error)
- func (ps Custom) Sort()
- func (ps Custom) Swap(i, j int)
- func (ps *Custom) UnmarshalBSON(in []byte) error
- type CustomPoint
- type Performance
- type PerformanceCounters
- type PerformanceCountersHDR
- type PerformanceGauges
- type PerformanceHDR
- type PerformanceTimers
- type PerformanceTimersHDR
- type Recorder
- func NewGroupedRecorder(collector ftdc.Collector, interval time.Duration) Recorder
- func NewHistogramGroupedRecorder(collector ftdc.Collector, interval time.Duration) Recorder
- func NewHistogramRecorder(collector ftdc.Collector) Recorder
- func NewIntervalHistogramRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder
- func NewIntervalRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder
- func NewRawRecorder(collector ftdc.Collector) Recorder
- func NewShimRecorder(r Recorder, tm TimerManager) Recorder
- func NewSingleHistogramRecorder(collector ftdc.Collector) Recorder
- func NewSingleRecorder(collector ftdc.Collector) Recorder
- func NewSynchronizedRecorder(r Recorder) Recorder
- type TimerManager
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Collector ¶
type Collector interface { AddEvent(*Performance) error ftdc.Collector }
Collector wraps the ftdc.Collector interface and adds specific awareness of the Performance type from this package. These collectors should be responsible for cumulative summing of values, when appropriate.
In general, implementations should obstruct calls to underlying collectors Add() method to avoid confusion, either by panicing or by no-oping.
func NewBasicCollector ¶
NewBasicCollector produces a collector implementation that adds Performance points to the underlying FTDC collector. Counter values in the point are added to the previous point so that the values are cumulative.
This event Collector implementation captures the maximal amount of fidelity and should be used except when dictated by retention strategy.
func NewIntervalCollector ¶
NewIntervalCollector constructs a Collector collapses events as in the other collector implementations, but will record at most a single event per interval.
func NewPassthroughCollector ¶
NewPassthroughCollector constructs a collector that does not sum Performance events and just passes them directly to the underlying collector.
func NewRandomSamplingCollector ¶
NewRandomSamplingCollector constructs a Collector that uses a psudorandom number generator (go's standard library math/rand) to select how often to record an event. All events are summed. Specify a percentage between 1 and 99 as the percent to reflect how many events to capture.
func NewSamplingCollector ¶
NewSamplingCollector constructs a collector that has the same semantics as the basic, adding all sampled documents together, but only persisting every n-th sample to the underlying collector.
func NewSynchronizedCollector ¶
NewSynchronizedCollector wraps another collector and wraps all required calls with the correct lock.
type Custom ¶
type Custom []CustomPoint
Custom is a collection of data points designed to store computed statistics at an interval. In general you will add a set of rolled-up data values to the custom object on an interval and then pass that sequence to an ftdc.Collector. Custom implements sort.Interface, and the CustomPoint type implements custom bson marshalling so that Points are marshaled as an object to facilitate their use with the ftdc format.
func MakeCustom ¶
MakeCustom creates a Custom slice with the specified size hint.
func (*Custom) Add ¶
Add appends a key to the Custom metric. Only accepts go native number types and timestamps.
func (Custom) MarshalBSON ¶
func (Custom) Sort ¶
func (ps Custom) Sort()
Sort is a convenience function around a stable sort for the custom array.
func (*Custom) UnmarshalBSON ¶
type CustomPoint ¶
type CustomPoint struct { Name string Value interface{} }
CustomPoint represents a computed statistic as a key value pair. Use with the Custom type to ensure that the Value types refer to number values and ensure consistent round trip semantics through BSON and FTDC.
type Performance ¶
type Performance struct { Timestamp time.Time `bson:"ts" json:"ts" yaml:"ts"` ID int64 `bson:"id" json:"id" yaml:"id"` Counters PerformanceCounters `bson:"counters" json:"counters" yaml:"counters"` Timers PerformanceTimers `bson:"timers" json:"timers" yaml:"timers"` Gauges PerformanceGauges `bson:"gauges" json:"gauges" yaml:"gauges"` }
Performance represents a single raw event in a metrics collection system for performance metric collection system.
Each point must report the timestamp of its collection.
func (*Performance) Add ¶
func (p *Performance) Add(in *Performance)
Add combines the values of the input Performance struct into this struct, logically, overriding the Gauges values as well as the timestamp and ID ID values, while summing the Counters and Timers values.
func (*Performance) MarshalBSON ¶
func (p *Performance) MarshalBSON() ([]byte, error)
MarshalBSON implements the bson marshaler interface to support converting this type into BSON without relying on a reflection-based BSON library.
func (*Performance) MarshalDocument ¶
func (p *Performance) MarshalDocument() (*birch.Document, error)
MarshalDocument exports the Performance type as a birch.Document to support more efficient operations.
func (*Performance) UnmarshalDocument ¶
func (p *Performance) UnmarshalDocument(doc *birch.Document) error
type PerformanceCounters ¶
type PerformanceCounters struct { Number int64 `bson:"n" json:"n" yaml:"n"` Operations int64 `bson:"ops" json:"ops" yaml:"ops"` Size int64 `bson:"size" json:"size" yaml:"size"` Errors int64 `bson:"errors" json:"errors" yaml:"errors"` }
PerformanceCounters refer to the number of operations/events or total of things since the last collection point. These values are used in computing various kinds of throughput measurements.
func (*PerformanceCounters) UnmarshalDocument ¶
func (p *PerformanceCounters) UnmarshalDocument(doc *birch.Document) error
type PerformanceCountersHDR ¶
type PerformanceGauges ¶
type PerformanceGauges struct { State int64 `bson:"state" json:"state" yaml:"state"` Workers int64 `bson:"workers" json:"workers" yaml:"workers"` Failed bool `bson:"failed" json:"failed" yaml:"failed"` }
PerformanceGauges holds simple counters that aren't expected to change between points, but are useful as annotations of the experiment or or descriptions of events in the system configuration.
func (*PerformanceGauges) UnmarshalDocument ¶
func (p *PerformanceGauges) UnmarshalDocument(doc *birch.Document) error
type PerformanceHDR ¶
type PerformanceHDR struct { Timestamp time.Time `bson:"ts" json:"ts" yaml:"ts"` ID int64 `bson:"id" json:"id" yaml:"id"` Counters PerformanceCountersHDR `bson:"counters" json:"counters" yaml:"counters"` Timers PerformanceTimersHDR `bson:"timers" json:"timers" yaml:"timers"` Gauges PerformanceGauges `bson:"guages" json:"guages" yaml:"guages"` }
PerformanceHDR the same as the Performance structure, but with all time duration and counter values stored as histograms.
func NewHistogramMillisecond ¶
func NewHistogramMillisecond(g PerformanceGauges) *PerformanceHDR
func NewHistogramSecond ¶
func NewHistogramSecond(g PerformanceGauges) *PerformanceHDR
func (*PerformanceHDR) MarshalDocument ¶
func (p *PerformanceHDR) MarshalDocument() (*birch.Document, error)
type PerformanceTimers ¶
type PerformanceTimers struct { Duration time.Duration `bson:"dur" json:"dur" yaml:"dur"` Total time.Duration `bson:"total" json:"total" yaml:"total"` }
PerformanceTimers refers to all of the timing data for this event. In general Total should equal the time since the last data point.
func (*PerformanceTimers) UnmarshalDocument ¶
func (p *PerformanceTimers) UnmarshalDocument(doc *birch.Document) error
type PerformanceTimersHDR ¶
type Recorder ¶
type Recorder interface { // The Inc<> operations add values to the specified counters tracked by // the collector. There is an additional "iteration" counter that the // recorder tracks based on the number of times that // BeginIteration/EndIteration are called, but is also accessible via // the IncIteration counter. // // In general, IncOperations should refer to the number of logical // operations collected. This differs from the iteration count, in the // case of workloads that comprise of multiple logical operations. // // Use IncSize to record, typically, the number of bytes processed or // generated by the operation. Use this in combination with logical // operations to be able to explore the impact of data size on overall // performance. Finally use IncError to track the number of errors // encountered during the event. IncIterations(int64) IncOperations(int64) IncError(int64) IncSize(int64) // The Set<> operations replace existing values for the state, workers, // and failed gauges. Workers should typically report the number of // active threads. The meaning of state depends on the test // but can describe phases of an experiment or operation. Use SetFailed // to flag a test as failed during the operation. SetWorkers(int64) SetState(int64) SetFailed(bool) // The BeginIteration and EndIteration methods mark the beginning and // end of a test's iteration. Typically calling EndIteration records // the duration specified as its argument and increments the counter // for number of iterations. Additionally there is a "total duration" // value captured which represents the total time taken in the // iteration in addition to the operation latency. // // The EndTest method writes any unpersisted material if the // collector's EndTest method has not. In all cases EndTest reports all // errors since the last EndTest call, and resets the internal error // the internal error tracking and unsets the tracked starting time. // Generally you should call EndTest once at the end of every test run, // and fail if there are errors reported. Reset does the same as // EndTest, except for persisting data and returing errors. BeginIteration() EndIteration(time.Duration) EndTest() error Reset() // SetID sets the unique id for the event, to allow users to identify // events per thread. SetID(int64) // SetTime defines the timestamp of the current point. SetTime is // usually not needed: BeginIteration will set the time to the current // time; however, if you're using a recorder as part of // post-processing, you will want to use SetTime directly. SetTime(time.Time) // SetTotalDuration allows you to set the total time covered by the // event in question. The total time is usually derived by the // difference between the time set in BeginIteration and the time when // EndTest is called. Typically the duration passed to EndTest refers // to a subset of this time (i.e. the amount of time that the // operations in question took), and the total time, includes some // period of overhead. // // In simplest terms, this should typically be the time since the last // event was recorded. SetTotalDuration(time.Duration) // SetDuration allows you to define the duration of a the operation, // this is likely a subset of the total duration, with the difference // between the duration and the total duration, representing some kind // of operational overhead. SetDuration(time.Duration) }
Recorder describes an interface that tests can use to track metrics and events during performance testing or normal operation. Implementations of recorder wrap an FTDC collector and will write data out to the collector for reporting purposes. The types produced by the collector use the Performance or PerformanceHDR types in this package.
Choose the implementation of Recorder that will capture all required data used by your test with sufficient resolution for use later. Additionally, consider the data volume produced by the recorder.
func NewGroupedRecorder ¶
NewGroupedRecorder blends the single and the interval recorders, but it persists during the EndIteration call only if the specified interval has elapsed. EndTest will persist any left over data.
The Group recorder is not safe for concurrent access without a synchronized wrapper.
func NewHistogramGroupedRecorder ¶
NewHistogramGroupedRecorder captures data and stores them with a histogram format. Like the Grouped recorder, it persists an event if the specified interval has elapsed since the last time an event was captured. The reset method also resets the last-collected time.
The timer histgrams have a minimum value of 1 microsecond, and a maximum value of 20 minutes, with 5 significant digits. The counter histograms store between 0 and 1 million, with 5 significant digits. The gauges are stored as integers.
The histogram Grouped reporter is not safe for concurrent use without a synchronixed wrapper.
func NewHistogramRecorder ¶
NewHistogramRecorder collects data and stores them with a histogram format. Like the Raw recorder, the system saves each data point after each call to EndIteration.
The timer histgrams have a minimum value of 1 microsecond, and a maximum value of 20 minutes, with 5 significant digits. The counter histograms store between 0 and 1 million, with 5 significant digits. The gauges are stored as integers.
The histogram reporter is not safe for concurrent use without a synchronized wrapper.
func NewIntervalHistogramRecorder ¶
func NewIntervalHistogramRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder
NewIntervalHistogramRecorder has similar semantics to histogram Grouped recorder, but has a background process that persists data on the specified on the specified interval rather than as a side effect of the EndTest call.
The background thread is started if it doesn't exist in the BeginIteration operation and is terminated by the EndTest operation.
The interval histogram recorder is safe for concurrent use.
func NewIntervalRecorder ¶
func NewIntervalRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder
NewIntervalRecorder has similar semantics to histogram Grouped recorder, but has a background process that persists data on the specified on the specified interval rather than as a side effect of the EndTest call.
The background thread is started in the BeginIteration operation if it does not already exist and is terminated by the EndTest operation.
The interval recorder is safe for concurrent use.
func NewRawRecorder ¶
NewRawRecorder records a new event every time that the EndIteration method is called.
The Raw recorder is not safe for concurrent access without a synchronized wrapper.
func NewShimRecorder ¶
func NewShimRecorder(r Recorder, tm TimerManager) Recorder
NewShimRecorder takes a recorder and acts as a thin recorder, using the TimeManager interface for relevant Begin & End values.
Go's standard library testing package has a *B type for benchmarking that can pass as a TimerManager.
func NewSingleHistogramRecorder ¶
NewSingleHistogramRecorder collects data and stores them with a histogram format. Like the Single recorder, the implementation persists the histogram every time you call EndTest.
The timer histograms have a minimum value of 1 microsecond, and a maximum value of 1 minute, with 5 significant digits. The counter histograms store store between 0 and 10 thousand, with 5 significant digits. The gauges are stored as integers.
The histogram Single reporter is not safe for concurrent use without a synchronized wrapper.
func NewSingleRecorder ¶
NewSingleRecorder records a single event every time the EndTest method is called, and otherwise just adds all counters and timing information to the underlying point.
The Single recorder is not safe for concurrent access without a synchronized wrapper.
func NewSynchronizedRecorder ¶
NewSynchronizedRecorder wraps a recorder implementation that is not concurrent safe in a recorder implementation that provides safe concurrent access without modifying the semantics of the recorder.
Most Recorder implementations are not safe for concurrent use, although some have this property as a result of persisting data on an interval.
type TimerManager ¶
type TimerManager interface { ResetTimer() StartTimer() StopTimer() }
TimerManager is a subset of the testing.B tool, used to manage setup code.
Source Files
¶
- collector.go
- custom.go
- histogram.go
- performance.go
- recorder.go
- recorder_histogram.go
- recorder_histogram_grouped.go
- recorder_histogram_interval.go
- recorder_histogram_single.go
- recorder_performance_grouped.go
- recorder_performance_interval.go
- recorder_performance_raw.go
- recorder_performance_single.go
- recorder_wrapper_stdlib.go
- recorder_wrapper_sync.go