huaweicloudcesreceiver

README

Huawei Cloud CES Receiver

Status
Stability	development: metrics
Distributions	[]
Issues
Code Owners	@heitorganzeli, @narcis96, @mwear

This receiver contains the implementation of the Huawei Cloud Cloud Eye Service (CES) receiver for the OpenTelemetry Collector. The receiver collects metrics from Huawei Cloud's CES service and sends them to the OpenTelemetry Collector for processing and exporting.

Configuration

The following settings are required:

• region_id: The ID of the Huawei Cloud region from which metrics are collected. For example, eu-west-101. The full list of the available regions can be found here.

• project_id: The ID of the project in Huawei Cloud. This is used to identify which project's metrics are to be collected. See Obtaining a Project ID.

• period: The aggregation granularity of metrics retrieved from CES in seconds. For details about the aggregation, see What Is Rollup?. Possible values are 1, 300, 1200, 3600, 14400, and 86400.

  • 1: Cloud Eye performs no aggregation and displays raw data.
  • 300: Cloud Eye aggregates data every 5 minutes.
  • 1200: Cloud Eye aggregates data every 20 minutes.
  • 3600: Cloud Eye aggregates data every hour.
  • 14400: Cloud Eye aggregates data every 4 hours.
  • 86400: Cloud Eye aggregates data every 24 hours.

• filter: The filter field determines the aggregation method used for the metrics. This aggregation is applied to the data points within the specified period. Valid values for filter include:

  • average: Calculates the average value over the period.
  • min: Retrieves the minimum value within the period.
  • max: Retrieves the maximum value within the period.
  • sum: Computes the sum of all data points within the period.
  • variance: Calculates the variance (square of the standard deviation) for the data points.

• no_verify_ssl: A boolean flag indicating whether SSL verification should be disabled. Set to True to disable SSL verification.

• access_key: The access key needed for CES authentification. Check Huawei Cloud SDK Authentication Setup section for more details.

• secret_key: The secret key needed for CES authentification. Check Huawei Cloud SDK Authentication Setup section for more details.

The following settings are optional:

• initial_delay: The delay before the first collection of metrics begins. This is a duration field, such as 5s for 5 seconds.

• collection_interval (default = 60s): This is the interval at which this receiver collects metrics. This value must be a string readable by Golang's time.ParseDuration. Valid time units are ns, us (or µs), ms, s, m, h. We recommend a polling interval of at least one minute.

• retry_on_failure: The following configurations can be used to control the retry policy of the CES client. The default values are suitable for most deployment scenarios.

  • enabled (default true)
  • initial_interval (default 100ms)
  • max_interval (default 1s)
  • max_elapsed_time (default 15s)
  • randomization_factor (default 0.5)
  • multiplier (default 1.5)

Example Configuration

receivers:
  huaweicloudcesreceiver:
    collection_interval: 3h
    initial_delay: 5s
    region_id: eu-west-101
    access_key: ${env:HUAWEICLOUD_SDK_AK}
    secret_key: ${env:HUAWEICLOUD_SDK_SK}
    project_id: "project_1"
    period: 300
    filter: average
    no_verify_ssl: True

The full list of settings exposed for this receiver are documented here.

Huawei Cloud SDK Authentication Setup

To ensure secure authentication, the Access Key (AK) and Secret Key (SK) used by the Huawei Cloud SDK must be stored in environment variables. See Obtaining an AK/SK.

Before running the application, you need to set the environment variables HUAWEICLOUD_SDK_AK and HUAWEICLOUD_SDK_SK in your local environment. Here’s how you can do it:

1. Open your terminal.

2. Set the environment variables by executing the following commands:

   export HUAWEICLOUD_SDK_AK=your-access-key
   export HUAWEICLOUD_SDK_SK=your-secret-key
   

3. Verify that the variables are set correctly:

   echo $HUAWEICLOUD_SDK_AK
   echo $HUAWEICLOUD_SDK_SK
   

Error handling

If you encounter any errors, please refer to:

• Huawei Cloud Error Codes
• CES Error Codes
• Quota management

Converting CES metric representation to Open Telementery metric representation

Source Field	Target Field	Description
metric_name	scoped_metric.metric.name	The name of the metric.
unit	scoped_metric.metric.unit	The unit of the metric, e.g., %, bit/s.
datapoints.value	scoped_metric.metric.gauge.data_points.value	The value of the metric at the specified timestamp.
datapoints.timestamp	scoped_metric.metric.gauge.data_points.timestamp	The timestamp of the metric in nanoseconds. Converted from milliseconds by multiplying by 1,000,000.
dimensions.name	key of scoped_metric.metric.metadata	The dimension name stored as an attribute.
dimensions.value	value of scoped_metric.metric.metadata	The dimension value stored as an attribute.
Receiver Config	resource.attributes["project.id"]	The project ID used in the configuration file of the receiver.
Receiver Config	resource.attributes["region.id"]	The region ID used in the configuration file of the receiver.
namespace	resource.attributes["service.namespace"]	The namespace of the metric, e.g., SYS.VPC, stored as an attribute with the key service.namespace.
N/A	resource.attributes["cloud.provider"]	Set to "huawei_cloud" as the cloud provider.
N/A	scoped_metric.scope.name	Set to "huawei_cloud_ces" as the scope name.
N/A	scoped_metric.scope.version	Set to "v1" as the scope version.

|

Notes

• The timestamp field in the source is converted from milliseconds to nanoseconds in the target field.
• Some fields are added in the target format with constant values to provide additional context and metadata.

Example:

[
    {
      "unit": "%",
      "datapoints": [
        { "average": 10, "timestamp": 1722580500000 },
        { "average": 20, "timestamp": 1722580800000 }
      ],
      "namespace": "SYS.ECS",
      "metric_name": "cpu_util",
      "dimensions": [
        {
          "name": "instance_id",
          "value": "faea5b75-e390-4e2b-8733-9226a9026070"
        }
      ]
    },
    {
      "unit": "%",
      "datapoints": [
        { "average": 30, "timestamp": 1722580500000 },
        { "average": 40, "timestamp": 1722580800000 }
      ],
      "namespace": "SYS.ECS",
      "metric_name": "mem_util",
      "dimensions": [
        {
          "name": "instance_id",
          "value": "abcea5b75-e390-4e2b-8733-9226a9026070"
        }
      ]
    },
    {
      "unit": "bit/s",
      "datapoints": [
        { "average": 1024, "timestamp": 1722580500000},
        { "average": 2048, "timestamp": 1722580800000}
      ],
      "namespace": "SYS.VPC",
      "metric_name": "upstream_bandwidth_usage",
      "dimensions": [
        {
          "name": "publicip_id",
          "value": "test-baae-4dd9-ad3f-1234"
        }
      ]
    }
]

converts to

{
  "resourceMetrics": [
    {
      "resource": {
        "attributes": {
            "cloud.provider": "huawei_cloud",
            "project.id": "project_1",
            "region.id": "eu-west-101",
            "system.namespace": "SYS.ECS",
          }
      },
      "scopeMetrics": [
        {
          "scope": { "name": "huawei_cloud_ces", "version": "v1" },
          "metrics": [
            {
              "name": "cpu_util",
              "unit": "%",
              "gauge": {
                "dataPoints": [
                  { "timeUnixNano": "1722580500000000000", "asDouble": 10 },
                  { "timeUnixNano": "1722580800000000000", "asDouble": 20 }
                ]
              },
              "metadata": {
                    "instance_id":  "faea5b75-e390-4e2b-8733-9226a9026070",
                }
            }
          ]
        },
        {
          "scope": { "name": "huawei_cloud_ces", "version": "v1" },
          "metrics": [
            {
              "name": "mem_util",
              "unit": "%",
              "gauge": {
                "dataPoints": [
                  { "timeUnixNano": "1722580500000000000", "asDouble": 30 },
                  { "timeUnixNano": "1722580800000000000", "asDouble": 40 }
                ]
              },
              "metadata": {
                  "instance_id": "abcea5b75-e390-4e2b-8733-9226a9026070",
              }
            }
          ]
        }
      ]
    },
    {
      "resource": {
        "attributes": {
            "cloud.provider": "huawei_cloud",
            "project.id": "project_1",
            "region.id": "eu-west-101",
            "system.namespace": "SYS.VPC",
          }
      },
      "scopeMetrics": [
        {
          "scope": { "name": "huawei_cloud_ces", "version": "v1" },
          "metrics": [
            {
              "name": "upstream_bandwidth_usage",
              "unit": "bits/s",
              "gauge": {
                "dataPoints": [
                  { "timeUnixNano": "1722580500000000000", "asDouble": 1024 },
                  { "timeUnixNano": "1722580800000000000", "asDouble": 2048 }
                ]
              },
              "metadata": {
                    "publicip_id":  "test-baae-4dd9-ad3f-1234",
                }
            }
          ]
        }
      ]
    }
  ]
}

Documentation

Index

• func NewFactory() receiver.Factory
• type Config

Functions

func NewFactory

func NewFactory() receiver.Factory

Types

type Config

type Config struct {
	scraperhelper.ControllerConfig `mapstructure:",squash"`
	confighttp.ClientConfig        `mapstructure:",squash"`
	// Nodes defines the nodes to scrape.
	// See https://www.elastic.co/guide/en/elasticsearch/reference/7.9/cluster.html#cluster-nodes for which selectors may be used here.
	// If Nodes is empty, no nodes will be scraped.
	Nodes []string `mapstructure:"nodes"`
	// SkipClusterMetrics indicates whether cluster level metrics from /_cluster/* endpoints should be scraped or not.
	SkipClusterMetrics bool `mapstructure:"skip_cluster_metrics"`
	// Indices defines the indices to scrape.
	// See https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#index-stats-api-path-params
	// for which names are viable.
	// If Indices is empty, no indices will be scraped.
	Indices []string `mapstructure:"indices"`

	// ProjectID is a string to reference project where metrics should be associated with.
	// If ProjectID is not filled in, the SDK will automatically call the IAM service to query the project id corresponding to the region.
	ProjectID string `mapstructure:"project_id"`

	// RegionID is the ID of the CES region.
	RegionID string `mapstructure:"region_id"`

	// How retrieved data from Cloud Eye is aggregated.
	// Possible values are 1, 300, 1200, 3600, 14400, and 86400.
	// 1: Cloud Eye performs no aggregation and displays raw data.
	// 300: Cloud Eye aggregates data every 5 minutes.
	// 1200: Cloud Eye aggregates data every 20 minutes.
	// 3600: Cloud Eye aggregates data every hour.
	// 14400: Cloud Eye aggregates data every 4 hours.
	// 86400: Cloud Eye aggregates data every 24 hours.
	// For details about the aggregation, see https://support.huaweicloud.com/intl/en-us/ces_faq/ces_faq_0009.html
	Period int32 `mapstructure:"period"`

	// Data aggregation method. The supported values ​​are max, min, average, sum, variance.
	Filter string `mapstructure:"filter"`

	BackOffConfig configretry.BackOffConfig `mapstructure:"retry_on_failure"`
	// contains filtered or unexported fields
}

Config represent a configuration for the CloudWatch logs exporter.