clientstats

README

Client stats reporting

Specification

The request JSON object is a non-nested object with the following properties. The process refers to which process data is associated with the request.

Property	Type	Process	Source	Description
version	int	Every request	hard-coded	Stats data specification version, current only 1 is accepted
timestamp	long	Every request	system time	Unix timestamp in milliseconds
process	string (enum)	Every request	hard-coded / cli flag	Enum values: validator, beaconnode, system
cpu_process_seconds_total	long	beaconnode, validator	prom: process_cpu_seconds_total	CPU seconds consumed by the process
memory_process_bytes	long	beaconnode, validator	prom: process_resident_memory_bytes	Number of bytes allocated to the process
client_name	string	beaconnode, validator	hard-coded to "prysm"	Name of the client type. Ex: prysm, lighthouse, nimbus, teku
client_version	string	beaconnode, validator	prom: prysm_version (label: version)	Client version. Ex: 1.0.0-beta.0
client_build	int	beaconnode, validator	prom: prysm_version (label: buildDate)	Integer representation of build for easier comparison
disk_beaconchain_bytes_total	long	beaconchain	prom: bcnode_disk_beaconchain_bytes_total	The amount of data consumed on disk by the beacon chain's database.
network_libp2p_bytes_total_receive	long	beaconchain	(currently unsupported)	The number of bytes received via libp2p traffic
network_libp2p_bytes_total_transmit	long	beaconchain	(currently unsupported)	The number of bytes transmitted via libp2p traffic
network_peers_connected	int	beaconchain	(currently unsupported)	The number of peers currently connected to the beacon chain
sync_eth1_connected	bool	beaconchain	prom: powchain_sync_eth1_connected	Whether or not the beacon chain node is connected to a synced eth1 node
sync_eth2_synced	bool	beaconchain	prom: beacon_clock_time_slot (true if this equals prom: beacon_head_slot)	Whether or not the beacon chain node is in sync with the beacon chain network
sync_beacon_head_slot	long	beaconchain	prom: beacon_head_slot	The head slot number.
sync_eth1_fallback_configured	bool	beaconchain	prom: powchain_sync_eth1_fallback_configured	Whether or not the beacon chain node has a fallback eth1 endpoint configured.
sync_eth1_fallback_connected	bool	beaconchain	prom: powchain_sync_eth1_fallback_connected	Whether or not the beacon chain node is connected to a fallback eth1 endpoint. A true value indicates a failed or interrupted connection with the primary eth1 endpoint.
slasher_active	bool	beaconchain	(coming soon)	Whether or not slasher functionality is enabled.
sync_eth2_fallback_configured	bool	validator	(currently unsupported)	Whether or not the process has a fallback eth2 endpoint configured
sync_eth2_fallback_connected	bool	validator	(currently unsupported)	Weather or not the process has connected to the failover eth2 endpoint. A true value indicates a failed or interrupted connection with the primary eth2 endpoint.
validator_total	int	validator	prom: validator_statuses (count of all peers)	The number of validating keys in use.
validator_active	int	validator	prom: validator_statuses (count of peers w/ "ACTIVE" status label)	The number of validator keys that are currently active.
cpu_cores	int	system	(currently unsupported)	The number of CPU cores available on the host machine
cpu_threads	int	system	(currently unsupported)	The number of CPU threads available on the host machine
cpu_node_system_seconds_total	long	system	(currently unsupported)	Overall CPU seconds observed on the host machine for all processes.
cpu_node_user_seconds_total	long	system	(currently unsupported)	??
cpu_node_iowait_seconds_total	long	system	(currently unsupported)	??
cpu_node_idle_seconds_total	long	system	(currently unsupported)	??
memory_node_bytes_total	long	system	(currently unsupported)	??
memory_node_bytes_free	long	system	(currently unsupported)	??
memory_node_bytes_cached	long	system	(currently unsupported)	??
memory_node_bytes_bufferd	long	system	(currently unsupported)	??
disk_node_bytes_total	long	system	(currently unsupported)	??
disk_node_bytes_free	long	system	(currently unsupported)	??
disk_node_io_seconds	long	system	(currently unsupported)	??
disk_node_reads_total	long	system	(currently unsupported)	??
disk_node_writes_total	long	system	(currently unsupported)	??
network_node_bytes_total_receive	long	system	(currently unsupported)	??
network_node_bytes_total_transmit	long	system	(currently unsupported)	??
misc_node_boot_ts_system	long	system	(currently unsupported)	??
misc_os	string	system	(currently unsupported)	Enum values: lin, win, mac, unk

The client stats reporter will submit a request object for each process type. The report request may submit a list of data or a single JSON object.

Examples

POST https://beaconcha.in/api/v1/stats/$API_KEY/$MACHINE_NAME

Single object payload

{
   "version": 1,
   "timestamp": 11234567,
   "process": "validator",
   "cpu_process_seconds_total": 1234567,
   "memory_process_bytes": 654321,
   "client_name": "lighthouse",
   "client_version": "1.1.2",
   "client_build": 12,
   "sync_eth2_fallback_configured": false,
   "sync_eth2_fallback_connected": false,
   "validator_total": 3,
   "validator_active": 2
}

Multiple object payload

[
   {
  	"version":1,
  	"timestamp":1618835497239,
  	"process":"beaconnode",
  	"cpu_process_seconds_total":6925,
  	"memory_process_bytes":1175138304,
  	"client_name":"lighthouse",
  	"client_version":"1.1.3",
  	"client_build":42,
  	"sync_eth2_fallback_configured":false,
  	"sync_eth2_fallback_connected":false,
  	"validator_active":1,
  	"validator_total":1
   },
   {
  	"version":1,
  	"timestamp":1618835497258,
  	"process":"system",
  	"cpu_cores":4,
  	"cpu_threads":8,
  	"cpu_node_system_seconds_total":1953818,
  	"cpu_node_user_seconds_total":229215,
  	"cpu_node_iowait_seconds_total":3761,
  	"cpu_node_idle_seconds_total":1688619,
  	"memory_node_bytes_total":33237434368,
  	"memory_node_bytes_free":500150272,
  	"memory_node_bytes_cached":13904945152,
  	"memory_node_bytes_buffers":517832704,
  	"disk_node_bytes_total":250436972544,
  	"disk_node_bytes_free":124707479552,
  	"disk_node_io_seconds":0,
  	"disk_node_reads_total":3362272,
  	"disk_node_writes_total":47766864,
  	"network_node_bytes_total_receive":26546324572,
  	"network_node_bytes_total_transmit":12057786467,
  	"misc_node_boot_ts_seconds":1617707420,
  	"misc_os":"unk"
   }
]

Documentation

Index

• Constants
• type APIMessage
• type BeaconNodeStats
• type CommonStats
• type Scraper
  • func NewBeaconNodeScraper(promExpoURL string) Scraper
  • func NewValidatorScraper(promExpoURL string) Scraper
• type Updater
  • func NewClientStatsHTTPPostUpdater(u string) Updater
  • func NewGenericClientStatsUpdater(w io.Writer) Updater
• type ValidatorStats

Constants

const (
	ClientName            = "prysm"
	BeaconNodeProcessName = "beaconnode"
	ValidatorProcessName  = "validator"
	APIVersion            = 1
)

Types

type APIMessage

type APIMessage struct {
	APIVersion  int    `json:"version"`
	Timestamp   int64  `json:"timestamp"` // unix timestamp in milliseconds
	ProcessName string `json:"process"`   // validator, beaconnode, system
}

APIMessage are common to all requests to the client-stats API Note that there is a "system" type that we do not currently support -- if we did APIMessage would be present on the system messages as well as validator and beaconnode, whereas CommonStats would only be part of beaconnode and validator.

type BeaconNodeStats

type BeaconNodeStats struct {
	// TODO(#8850): add support for this after slasher refactor is merged
	SlasherActive             bool  `json:"slasher_active"`
	SyncEth1Connected         bool  `json:"sync_eth1_connected"`
	SyncEth2Synced            bool  `json:"sync_eth2_synced"`
	DiskBeaconchainBytesTotal int64 `json:"disk_beaconchain_bytes_total"`
	// N/A -- would require significant network code changes at this time
	NetworkLibp2pBytesTotalReceive int64 `json:"network_libp2p_bytes_total_receive"`
	// N/A -- would require significant network code changes at this time
	NetworkLibp2pBytesTotalTransmit int64 `json:"network_libp2p_bytes_total_transmit"`
	// p2p_peer_count where label "state" == "Connected"
	NetworkPeersConnected int64 `json:"network_peers_connected"`
	// beacon_head_slot
	SyncBeaconHeadSlot int64 `json:"sync_beacon_head_slot"`
	CommonStats        `json:",inline"`
}

BeaconNodeStats embeds CommonStats and represents metrics specific to the beacon-node process. This type is used to marshal metrics data to the POST body sent to the metrics collcetor. To make the connection to client-stats clear, BeaconNodeStats is also used by prometheus collection code introduced to support client-stats. Note that some metrics are labeled NA because they are expected to be present with their zero-value when not supported by a client.

type CommonStats

type CommonStats struct {
	CPUProcessSecondsTotal int64  `json:"cpu_process_seconds_total"`
	MemoryProcessBytes     int64  `json:"memory_process_bytes"`
	ClientName             string `json:"client_name"`
	ClientVersion          string `json:"client_version"`
	ClientBuild            int64  `json:"client_build"`
	// TODO(#8849): parse the grpc connection string to determine
	// if multiple addresses are present
	SyncEth2FallbackConfigured bool `json:"sync_eth2_fallback_configured"`
	// N/A -- when multiple addresses are provided to grpc, requests are
	// load-balanced between the provided endpoints.
	// This is different from a "fallback" configuration where
	// the second address is treated as a failover.
	SyncEth2FallbackConnected bool `json:"sync_eth2_fallback_connected"`
	APIMessage                `json:",inline"`
}

CommonStats represent generic metrics that are expected on both beaconnode and validator metric types. This type is used for marshaling metrics to the POST body sent to the metrics collcetor. Note that some metrics are labeled NA because they are expected to be present with their zero-value when not supported by a client.

type Scraper

type Scraper interface {
	Scrape() (io.Reader, error)
}

A Scraper polls the data source it has been configured with and interprets the content to produce a client-stats process metric. Scrapers currently exist to produce 'validator' and 'beaconnode' metric types.

func NewBeaconNodeScraper

func NewBeaconNodeScraper(promExpoURL string) Scraper

NewBeaconNodeScraper constructs a Scaper capable of scraping the prometheus endpoint of a beacon-node process and producing the json body for the beaconnode client-stats process type.

func NewValidatorScraper

func NewValidatorScraper(promExpoURL string) Scraper

NewValidatorScraper constructs a Scaper capable of scraping the prometheus endpoint of a validator process and producing the json body for the validator client-stats process type.

type Updater

type Updater interface {
	Update(io.Reader) error
}

An Updater can take the io.Reader created by Scraper and send it to a data sink for consumption. An Updater is used for instance ot send the scraped data for a beacon-node to a remote client-stats endpoint.

func NewClientStatsHTTPPostUpdater

func NewClientStatsHTTPPostUpdater(u string) Updater

NewClientStatsHTTPPostUpdater is used when the update endpoint is reachable via an HTTP POST request.

func NewGenericClientStatsUpdater

func NewGenericClientStatsUpdater(w io.Writer) Updater

NewGenericClientStatsUpdater can Update any io.Writer. It is used by the cli to write to stdout when an http endpoint is not provided. The output could be piped into another program or used for debugging.

type ValidatorStats

type ValidatorStats struct {
	// N/A -- TODO(#8848): verify whether we can obtain this metric from the validator process
	ValidatorTotal int64 `json:"validator_total"`
	// N/A -- TODO(#8848): verify whether we can obtain this metric from the validator process
	ValidatorActive int64 `json:"validator_active"`
	CommonStats     `json:",inline"`
}

ValidatorStats embeds CommonStats and represents metrics specific to the validator process. This type is used to marshal metrics data to the POST body sent to the metrics collcetor. Note that some metrics are labeled NA because they are expected to be present with their zero-value when not supported by a client.