pulsar

README

Apache Pulsar monitoring with Netdata

Apache Pulsar is an open-source distributed pub-sub messaging system.

This module will monitor one or more Apache Pulsar instances, depending on your configuration.

It collects broker statistics from the prometheus endpoint.

pulsar module is tested on the following versions:

• v2.5.0

Metrics

All metrics have "pulsar." prefix.

• topic_* metrics are available when exposeTopicLevelMetricsInPrometheus is set to true.
• subscription_* and namespace_subscription metrics are available when exposeTopicLevelMetricsInPrometheus si set to true.
• replication_* and namespace_replication_* metrics are available when replication is configured and replicationMetricsEnabled is set to true

Metric	Scope	Dimensions	Units
broker_components	global	namespaces, topics, subscriptions, producers, consumers	components
messages_rate	global	publish, dispatch	messages/s
throughput_rate	global	publish, dispatch	KiB/s
storage_size	global	used	KiB
storage_operations_rate	global	read, write	message batches/s
msg_backlog	global	backlog	messages
storage_write_latency	global	<=0.5ms, <=1ms, <=5ms, =10ms, <=20ms, <=50ms, <=100ms, <=200ms, <=1s, >1s	entries/s
entry_size	global	<=128B, <=512B, <=1KB, <=2KB, <=4KB, <=16KB, <=100KB, <=1MB, >1MB	entries/s
subscription_delayed	global	delayed	message bacthes
subscription_msg_rate_redeliver	global	redelivered	messages/s
subscription_blocked_on_unacked_messages	global	blocked	subscriptions
replication_rate	global	in, out	messages/s
replication_throughput_rate	global	in, out	KiB/s
replication_backlog	global	backlog	messages
namespace_broker_components	global	topics, subscriptions, producers, consumers	components
namespace_messages_rate	namespace	publish, dispatch	messages/s
namespace_throughput_rate	namespace	publish, dispatch	KiB/s
namespace_storage_size	namespace	used	KiB
namespace_storage_operations_rate	namespace	read, write	message batches/s
namespace_msg_backlog	namespace	backlog	messages
namespace_storage_write_latency	namespace	<=0.5ms, <=1ms, <=5ms, =10ms, <=20ms, <=50ms, <=100ms, <=200ms, <=1s, >1s	entries/s
namespace_entry_size	namespace	<=128B, <=512B, <=1KB, <=2KB, <=4KB, <=16KB, <=100KB, <=1MB, >1MB	entries/s
namespace_subscription_delayed	namespace	delayed	message bacthes
namespace_subscription_msg_rate_redeliver	namespace	redelivered	messages/s
namespace_subscription_blocked_on_unacked_messages	namespace	blocked	subscriptions
namespace_replication_rate	namespace	in, out	messages/s
namespace_replication_throughput_rate	namespace	in, out	KiB/s
namespace_replication_backlog	namespace	backlog	messages
topic_producers	namespace	a dimension per topic	producers
topic_subscriptions	namespace	a dimension per topic	subscriptions
topic_consumers	namespace	a dimension per topic	consumers
topic_messages_rate_in	namespace	a dimension per topic	publishes/s
topic_messages_rate_out	namespace	a dimension per topic	dispatches/s
topic_throughput_rate_in	namespace	a dimension per topic	KiB/s
topic_throughput_rate_out	namespace	a dimension per topic	KiB/s
topic_storage_size	namespace	a dimension per topic	KiB
topic_storage_read_rate	namespace	a dimension per topic	message batches/s
topic_storage_write_rate	namespace	a dimension per topic	message batches/s
topic_msg_backlog	namespace	a dimension per topic	messages
topic_subscription_delayed	namespace	a dimension per topic	message batches
topic_subscription_msg_rate_redeliver	namespace	a dimension per topic	messages/s
topic_subscription_blocked_on_unacked_messages	namespace	a dimension per topic	blocked subscriptions
topic_replication_rate_in	namespace	a dimension per topic	messages/s
topic_replication_rate_out	namespace	a dimension per topic	messages/s
topic_replication_throughput_rate_in	namespace	a dimension per topic	messages/s
topic_replication_throughput_rate_out	namespace	a dimension per topic	messages/s
topic_replication_backlog	namespace	a dimension per topic	messages
topic_replication_backlog	namespace	a dimension per topic	messages

Configuration

Edit the go.d/pulsar.conf configuration file using edit-config from the Netdata config directory, which is typically at /etc/netdata.

cd /etc/netdata # Replace this path with your Netdata config directory
sudo ./edit-config go.d/pulsar.conf

Needs only url to server's /metrics endpoint. Here is an example for 2 servers:

jobs:
  - name: local
    url: http://127.0.0.1:8080/metrics

  - name: remote
    url: http://203.0.113.10:8080/metrics

For all available options please see module configuration file.

Topic filtering

By default, module collects data for all topics, but it supports topic filtering. Filtering doesn't exclude a topic stats from the summary/namespace stats, it only removes the topic from the topic charts.

To check matcher syntax see matcher documentation.

  - name: local
    url: http://127.0.0.1:8080/metrics
    topic_filter:
      includes:
        - matcher1
        - matcher2
      excludes:
        - matcher1
        - matcher2

Update every

Module default update_every is 60.

Apache Pulsar doesnt expose raw counters, it exposes rate. It counts rates every statsUpdateFrequencyInSecs. Default value is 60 seconds.

Module update_every should be equal to statsUpdateFrequencyInSecs.

Troubleshooting

To troubleshoot issues with the pulsar collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/
  

• Switch to the netdata user.

  sudo -u netdata -s
  

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m pulsar
  

Documentation

Index

• type Chart
• type Charts
• type Config
• type Dim
• type Dims
• type Opts
• type Pulsar
  • func New() *Pulsar
  • func (p *Pulsar) Charts() *Charts
  • func (p *Pulsar) Check() bool
  • func (Pulsar) Cleanup()
  • func (p *Pulsar) Collect() map[string]int64
  • func (p *Pulsar) Init() bool

Types

type Chart

type Chart = module.Chart

type Charts

type Charts = module.Charts

type Config

type Config struct {
	web.HTTP   `yaml:",inline"`
	TopicFiler matcher.SimpleExpr `yaml:"topic_filter"`
}

type Dim

type Dim = module.Dim

type Dims

type Dims = module.Dims

type Opts

type Opts = module.Opts

type Pulsar

type Pulsar struct {
	module.Base
	Config `yaml:",inline"`
	// contains filtered or unexported fields
}

func New

func New() *Pulsar

func (*Pulsar) Charts

func (p *Pulsar) Charts() *Charts

func (*Pulsar) Check

func (p *Pulsar) Check() bool

func (Pulsar) Cleanup

func (Pulsar) Cleanup()

func (*Pulsar) Collect

func (p *Pulsar) Collect() map[string]int64

func (*Pulsar) Init

func (p *Pulsar) Init() bool