Documentation
¶
Overview ¶
Package metrics implements the Beam metrics API, described at http://s.apache.org/beam-metrics-api
Metrics in the Beam model are uniquely identified by a namespace, a name, and the PTransform context in which they are used. Further, they are reported as a delta against the bundle being processed, so that overcounting doesn't occur if a bundle needs to be retried. Each metric is scoped to their bundle, and ptransform.
Cells (or metric cells) are defined for each Beam model metric type, and the serve as concurrency safe storage of a given metric's values. Proxys are exported values representing the metric, for use in user ptransform code. They don't retain their cells, since they don't have the context to be able to store them for export back to the pipeline runner.
Metric cells aren't initialized until their first mutation, which follows from the Beam model design, where metrics are only sent for a bundle if they have changed. This is particularly convenient for distributions which means their min and max fields can be set to the first value on creation rather than have some marker of uninitialized state, which would otherwise need to be checked for on every update.
Metric values are implemented as lightweight proxies of the user provided namespace and name. This allows them to be declared globally, and used in any ParDo. Further, as per the design, they can be declared dynamically at runtime.
To handle reporting deltas on the metrics by bundle, metrics are keyed by bundleID,PTransformID,namespace, and name, so metrics that are identical except for bundles are treated as distinct, effectively providing per bundle deltas, since a new value cell is used per bundle.
Index ¶
- func DumpToLog(ctx context.Context)
- func DumpToLogFromStore(ctx context.Context, store *Store)
- func DumpToOutFromContext(ctx context.Context)
- func DumpToOutFromStore(store *Store)
- func SetBundleID(ctx context.Context, id string) context.Context
- func SetPTransformID(ctx context.Context, id string) context.Context
- type Counter
- type Distribution
- type Extractor
- type Gauge
- type Labels
- type Store
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DumpToLog ¶
DumpToLog is a debugging function that outputs all metrics available locally to beam.Log.
func DumpToLogFromStore ¶
DumpToLogFromStore dumps the metrics in the provided Store to beam.Log.
func DumpToOutFromContext ¶
DumpToOutFromContext is a debugging function that outputs all metrics available locally to std out, extracting the metric store from the context.
func DumpToOutFromStore ¶
func DumpToOutFromStore(store *Store)
DumpToOutFromStore is a debugging function that outputs all metrics available locally to std out directly from the store.
func SetBundleID ¶
SetBundleID sets the id of the current Bundle, and populates the store.
Types ¶
type Counter ¶
type Counter struct {
// contains filtered or unexported fields
}
Counter is a simple counter for incrementing and decrementing a value.
func NewCounter ¶
NewCounter returns the Counter with the given namespace and name.
type Distribution ¶
type Distribution struct {
// contains filtered or unexported fields
}
Distribution is a simple distribution of values.
func NewDistribution ¶
func NewDistribution(ns, n string) *Distribution
NewDistribution returns the Distribution with the given namespace and name.
func (*Distribution) String ¶
func (m *Distribution) String() string
type Extractor ¶
type Extractor struct {
// SumInt64 extracts data from Sum Int64 counters.
SumInt64 func(labels Labels, v int64)
// DistributionInt64 extracts data from Distribution Int64 counters.
DistributionInt64 func(labels Labels, count, sum, min, max int64)
// GaugeInt64 extracts data from Gauge Int64 counters.
GaugeInt64 func(labels Labels, v int64, t time.Time)
}
Extractor allows users to access metrics programatically after pipeline completion. Users assign functions to fields that interest them, and that function is called for each metric of the associated kind.
func (Extractor) ExtractFrom ¶
ExtractFrom the given metrics Store all the metrics for populated function fields. Returns an error if no fields were set.
type Gauge ¶
type Gauge struct {
// contains filtered or unexported fields
}
Gauge is a time, value pair metric.
type Labels ¶
type Labels struct {
// contains filtered or unexported fields
}
Labels provide the context for the given metric.
func PCollectionLabels ¶
PCollectionLabels builds a Labels for pcollection metrics. Intended for framework use.
func PTransformLabels ¶
PTransformLabels builds a Labels for transform metrics. Intended for framework use.
func UserLabels ¶
UserLabels builds a Labels for user metrics. Intended for framework use.
Source Files