apachesparkreceiver

README

Apache Spark Receiver

Status
Stability	alpha: metrics
Distributions	contrib
Issues
Code Owners	@djaglowski, @Caleb-Hurshman, @mrsillydog

This receiver fetches metrics for an Apache Spark cluster through the Apache Spark REST API - specifically, the /metrics/json, /api/v1/applications/[app-id]/stages, /api/v1/applications/[app-id]/executors, and /api/v1/applications/[app-id]/jobs endpoints.

Purpose

The purpose of this component is to monitor Apache Spark clusters and the applications running on them through the collection of performance metrics like memory utilization, CPU utilization, shuffle operations, garbage collection time, I/O operations, and more.

Prerequisites

This receiver supports Apache Spark versions:

• 3.3.2+

Configuration

These configuration options are for connecting to an Apache Spark application.

The following settings are optional:

• collection_interval: (default = 60s): This receiver collects metrics on an interval. This value must be a string readable by Golang's time.ParseDuration. Valid time units are ns, us (or µs), ms, s, m, h.
• initial_delay (default = 1s): defines how long this receiver waits before starting.
• endpoint: (default = http://localhost:4040): Apache Spark endpoint to connect to in the form of [http][://]{host}[:{port}]
• application_names: An array of Spark application names for which metrics should be collected. If no application names are specified, metrics will be collected for all Spark applications running on the cluster at the specified endpoint.

Example Configuration

receivers:
  apachespark:
    collection_interval: 60s
    endpoint: http://localhost:4040
    application_names:
    - PythonStatusAPIDemo
    - PythonLR

The full list of settings exposed for this receiver are documented here with detailed sample configurations here.

Metrics

Details about the metrics produced by this receiver can be found in metadata.yaml

Documentation

Index

• func NewFactory() receiver.Factory
• type Config
  • func (cfg *Config) Validate() error

Functions

func NewFactory

func NewFactory() receiver.Factory

NewFactory creates a new receiver factory for Spark

Types

type Config

type Config struct {
	scraperhelper.ControllerConfig `mapstructure:",squash"`
	metadata.MetricsBuilderConfig  `mapstructure:",squash"`
	confighttp.ClientConfig        `mapstructure:",squash"`
	ApplicationNames               []string `mapstructure:"application_names"`
}

Config defines the configuration for the various elements of the receiver agent.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate validates missing and invalid configuration fields.