README
¶
heka-prometheus
mozilla heka output plugin which exposes an endpoint for prometheus to scrape. The internals are heavily influenced by the collectd_exporter
Metrics are created and registered with the Prometheus client using an immutable implementation of the Metric interface NewConstMetric
Status Expiremental
As advertised the internal message format has changed from a native Heka message to a json payload. Filters need to be able to emit 100's of metrics on timer_event. The author wasn't able to grok how to encode many metric events while still providing an expressive, albeit arbitrary, tagging of each metric on a single Heka message. Sending 100's of new messages from a single filter timer_event hung the router pretty badly.
The json payload solves that and also means external stuff can be blindly forwarded too.
Usage
Send a heka message to this output plugin w/ a json message in the Payload field.
The top level keys are one of single, histogram, and summary which translate to the different Constant Metrics types in prometheus.
Lists of each are sent as a subdocument of each key.
single requires the valuetype key which specifies counter or gauge.
expires specifies seconds the metric should survive. Expiration is calculated by adding expires to the message timestamp (heka has timestamps.)
Metrics lacking expires inherit from the default specified in toml.
entire body example:
{
"histogram": [
{
"Buckets": {
"100.1": 12
},
"count": 1,
"help": "history of stuff",
"labels": {
"period": "20th century"
},
"name": "hekademo_history1",
"sum": 100
}
],
"single": [
{
"help": "a counter that counts stuff",
"labels": {
"role": "barista", "shift": "morning"
},
"name": "hekademo_counter1",
"value": 10000.123,
"valuetype": "counter"
},
{
"expires": 100,
"help": "the gas tank",
"value": 0.123,
"labels": {
"car": "mine", "grade": "premium"
},
"name": "hekademo_gauge2",
"valuetype": "gauge"
}
],
"summary": [
{
"Count": 2,
"Quantiles": {
"50": 80,
"90": 20
},
"Sum": 100,
"help": "summary of stuff",
"name": "hekademo_summary1"
}
]
}
Add the following toml to heka:
[prometheus_out]
type = "PrometheusOutput"
message_matcher = 'Logger == "Anything"' # anything to route the message properly here
Address = "127.0.0.1:9112"
default_ttl = '15s' # applied to any metrics w/ no expires, defautls to 90s
curl the new prometheus in heka:
[david@foulplay ~]$ curl http://127.0.0.1:9112/metrics
# [lots of prometheus boilerplate metrics suprressed]
#
#
# HELP hekagateway_msg_failed improperly formatted messages
# # TYPE hekagateway_msg_failed counter
# hekagateway_msg_failed 3
# # HELP hekagateway_msg_success properly formatted messages
# # TYPE hekagateway_msg_success counter
# hekagateway_msg_success 0
## HELP net_counters number of packets on network
## TYPE net_counters gauge
#net_counters{host="machine1",service="webapp"} 100
Building
append the following to heka/cmake/plugin_loader.cmake
hg_clone(https://bitbucket.org/ww/goautoneg default)
git_clone(https://github.com/prometheus/client_golang master)
git_clone(https://github.com/prometheus/procfs master)
git_clone(http://github.com/matttproud/golang_protobuf_extensions master)
git_clone(http://github.com/golang/protobuf master)
git_clone(https://github.com/prometheus/client_model master)
git_clone(http://github.com/beorn7/perks master)
git_clone(http://github.com/pquerna/ffjson master)
add_external_plugin(git https://github.com/davidbirdsong/heka-promethus master)
Putting it All Together
a contrived example since most interesting data could come from within heka
minimal toml
[HttpListenInput]
address = "0.0.0.0:8325"
[prometheus_out]
type = "PrometheusOutput"
message_matcher = 'Type == "heka.httpdata.request"'
Address = "127.0.0.1:9112"
default_ttl = '15s' # applied to any metrics w/ no expire field
POST data to heka's listen port
[david@foulplay ~]$ curl -X POST http://127.0.0.1:8325 -d '{
"histogram": [
{
"Buckets": {
"100.1": 12
},
"count": 1,
"help": "history of stuff",
"labels": {
"period": "20th century"
},
"name": "hekademo_history1",
"sum": 100
}
],
"single": [
{
"help": "a counter that counts stuff",
"labels": {
"role": "barista", "shift": "morning"
},
"name": "hekademo_counter1",
"value": 10000.123,
"valuetype": "counter"
},
{
"expires": 100,
"help": "the gas tank",
"value": 0.123,
"labels": {
"car": "mine", "grade": "premium"
},
"name": "hekademo_gauge2",
"valuetype": "gauge"
}
],
"summary": [
{
"Count": 2,
"Quantiles": {
"50": 80,
"90": 20
},
"Sum": 100,
"help": "summary of stuff",
"name": "hekademo_summary1"
}
]
}
'
scrape heka for recent data
[david@foulplay ~]$ curl -s http://127.0.0.1:9112/metrics | grep -B2 hekademo
# TYPE go_goroutines gauge
go_goroutines 22
# HELP hekademo_counter1 a counter that counts stuff
# TYPE hekademo_counter1 counter
hekademo_counter1{role="barista",shift="morning"} 10000.123
# HELP hekademo_gauge2 the gas tank
# TYPE hekademo_gauge2 gauge
hekademo_gauge2{car="mine",grade="premium"} 0.123
# HELP hekademo_history1 history of stuff
# TYPE hekademo_history1 histogram
hekademo_history1_bucket{period="20th century",le="100.1"} 12
hekademo_history1_bucket{period="20th century",le="+Inf"} 1
hekademo_history1_sum{period="20th century"} 100
hekademo_history1_count{period="20th century"} 1
# HELP hekademo_summary1 summary of stuff
# TYPE hekademo_summary1 summary
hekademo_summary1{quantile="50"} 80
hekademo_summary1{quantile="90"} 20
hekademo_summary1_sum 100
hekademo_summary1_count 2
TODO
- multi types
SummaryandHistogrammight need more data validation
Documentation
¶
Overview ¶
Package prometheus implements a heka output plugin which provides acts as a prometheus endpoint ready for scraping
Messages that arrive via the heka router must have a carefully formatted structure, all data is conveyed in Heka Fields: http://hekad.readthedocs.org/en/v0.9.2/message/index.html
Prometheus Data types limited to: Gauge and GaugeVect /
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ConstHistogram ¶
type ConstHistogram struct {
Count uint64
Sum float64
Buckets map[string]uint64
Name string
Labels map[string]string
Help string
Expires int64
// contains filtered or unexported fields
}
func (*ConstHistogram) MarshalJSON ¶
func (mj *ConstHistogram) MarshalJSON() ([]byte, error)
func (*ConstHistogram) MarshalJSONBuf ¶
func (mj *ConstHistogram) MarshalJSONBuf(buf fflib.EncodingBuffer) error
func (*ConstHistogram) UnmarshalJSON ¶
func (uj *ConstHistogram) UnmarshalJSON(input []byte) error
func (*ConstHistogram) UnmarshalJSONFFLexer ¶
func (uj *ConstHistogram) UnmarshalJSONFFLexer(fs *fflib.FFLexer, state fflib.FFParseState) error
type ConstMetric ¶
type ConstMetric struct {
Value float64
ValueType string
Name string
Labels map[string]string
Help string
Expires int64
// contains filtered or unexported fields
}
func (*ConstMetric) MarshalJSON ¶
func (mj *ConstMetric) MarshalJSON() ([]byte, error)
func (*ConstMetric) MarshalJSONBuf ¶
func (mj *ConstMetric) MarshalJSONBuf(buf fflib.EncodingBuffer) error
func (*ConstMetric) UnmarshalJSON ¶
func (uj *ConstMetric) UnmarshalJSON(input []byte) error
func (*ConstMetric) UnmarshalJSONFFLexer ¶
func (uj *ConstMetric) UnmarshalJSONFFLexer(fs *fflib.FFLexer, state fflib.FFParseState) error
type ConstSummary ¶
type ConstSummary struct {
Count uint64
Sum float64
Quantiles map[string]float64
Name string
Labels map[string]string
Help string
Expires int64
// contains filtered or unexported fields
}
func (*ConstSummary) MarshalJSON ¶
func (mj *ConstSummary) MarshalJSON() ([]byte, error)
func (*ConstSummary) MarshalJSONBuf ¶
func (mj *ConstSummary) MarshalJSONBuf(buf fflib.EncodingBuffer) error
func (*ConstSummary) UnmarshalJSON ¶
func (uj *ConstSummary) UnmarshalJSON(input []byte) error
func (*ConstSummary) UnmarshalJSONFFLexer ¶
func (uj *ConstSummary) UnmarshalJSONFFLexer(fs *fflib.FFLexer, state fflib.FFParseState) error
type Metrics ¶
type Metrics struct {
Single []*ConstMetric
Summary []*ConstSummary
Histogram []*ConstHistogram
}
func (*Metrics) MarshalJSON ¶
func (*Metrics) MarshalJSONBuf ¶
func (mj *Metrics) MarshalJSONBuf(buf fflib.EncodingBuffer) error
func (*Metrics) UnmarshalJSON ¶
func (*Metrics) UnmarshalJSONFFLexer ¶
type PromOut ¶
type PromOut struct {
// contains filtered or unexported fields
}
func (*PromOut) Collect ¶
func (p *PromOut) Collect(ch chan<- prometheus.Metric)
func (*PromOut) ConfigStruct ¶
func (p *PromOut) ConfigStruct() interface{}
func (*PromOut) Describe ¶
func (p *PromOut) Describe(ch chan<- *prometheus.Desc)
func (*PromOut) Run ¶
func (p *PromOut) Run(or pipeline.OutputRunner, ph pipeline.PluginHelper) (err error)
Source Files