metrics

package
v0.1.10-64-g7180dea Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 27, 2019 License: MIT Imports: 22 Imported by: 0

README

Daemon Metrics & Monitoring Services

snet-daemon sends heartbeat upon the request from a monitoring service or any other component. Apart from heartbeat, it will expose few custom metrics as mentioned below metrics section.
The critical daemon updates, warning, or errors can be reported to a pre-configured email using a default notification service offered by SingularityNet. Its just an optional service. Service providers are free to choose thier own endpoints/web hooks for sending alerts.

Registration

Daemon Registration is required for the metrics services to uniquely identify every daemon and store the metrics accordingly. To register with metrics service, every daemon must have a Unique Identity i.e. DaemonID. Daemon ID is a SHA256 hash value generated by using - Org Id, Service ID, Group Id of Daemon (Derived from from service metadata) and Daemon Endpoint.

daemonID = SHA256(config.GetString(config.OrgnaizationId) + 
                  config.GetString(config.ServiceId) + 
                  group_Id + //Groupd Id of daemon
                  config.GetString(config.DaemonEndPoint)

Post beta, this ID will be used to enable Token based authentication for accessing metrics services.

Heartbeat

Heartbeat indicates the state of the Daemon and the service associated with it. The heartbeat message from the daemon wraps the heartbeat of the service and sends it when requested. Heartbeat is always pull based, i.e. The monitoring service will have to call the heartbeat service to get the Daemon state.

When the monitoring services calls daemon heartbeat, the daemon internally makes a call to service heartbeat endpoint. Upon receiving the service heartbeat, Daemon wraps it in the final heartbeat and sends it to the calling service.

For the service heartbeat, implementation followed the standard health checking protocol as defined in gRPC Health Checking Protocol. The service must use the same proto and implement the heartbeat functionality.

Heartbeats will be enabled by default and they cant be disabled.
Configuration

  • service_heartbeat_type (mandatory. http|grpc) - though All services must be using grpc, for the simplicity of heartbeat implementation, both http and gRPC based heartbeat end points are supported.

  • heartbeat_svc_end_point (mandatory must be a valid http|https|grpc url) - It is service heartbeat endpoint. Based on the service heart beat type, if the type is http, then heartbeat service endpoint must be a HTTP endpoint, similarly, if the type is gRPC, then the heartbeat service must follow the health protocol as mentioned above and the gRPC endpoint has to be configured here.

Service endpoints

Heartbeat service is exposed from Daemon endpoint itself, but with with different route {daemon_endpoint}/heartbeat It is pull based service i.e. the monitoring service must call this endpoint to get the heartbeat.

GET http://127.0.0.1:8080/heartbeat

Sample heartbeat from the daemon, which contains the service heartbeat as well

{
  "daemonID": "3a4ebeb75eace1857a9133c7a50bdbb841b35de60f78bc43eafe0d204e523dfe",
  "timestamp": "1544916260",
  "status": "Online",
  "serviceheartbeat": "{\"serviceID\":\"sample1\", \"status\":\"SERVING\"}"
}

Daemon must call the services as configured in heartbeat_svc_end_point and the type to get the service heartbeat. Sample Heartbeat Service result is

GET http://127.0.0.1:25000/heartbeat

{
  "serviceID":"sample1", 
  "status":"SERVING"
}
Daemon Monitoring

Each incoming request, outgoing response will be intercepted and the corresponding metrics will be extracted. The extracted metrics will be reported immediately to the metrics services as configured in the Daemon configuration.

Metrics collection is enabled by default in configuration. Service Provider can disable the metrics anytime, if he/she doesn wanted to collect any metrics.

Sample metrics being collected are as below

{
   "request_id": "bggd8ipod0kv0c9a3fs0",
   "input_data_size": 8,
   "content-type": "application/grpc",
   "service_method": "/example_service.Calculator/div",
   "user-agent": "grpc-python/1.17.1 grpc-c/7.0.0 (manylinux; chttp2; gizmo)",
   "request_received_time": "2018-12-24T12:42:51Z",
   "organization_id": "ExampleOrganizationID",
   "service_id": "ExampleServiceID",
   "Group_id": "B6r6a/TvJ36SvOrvyZHxQtDJDYNmWm3Y1/tqhJrKqFM=",
   "Daemon_end_point": "localhost:8080"
 }

Sample Payload for response stats


{
  "request_id": "bggdcdhod0kv0c9a3ft0",
  "organization_id": "ExampleOrganizationID",
  "service_id": "ExampleServiceID",
  "Group_id": "B6r6a/TvJ36SvOrvyZHxQtDJDYNmWm3Y1/tqhJrKqFM=",
  "Daemon_end_point": "localhost:8080",
  "response_sent_time": "2018-12-24T12:59:51Z",
  "response_time": "23.724177879s",
  "response_code": "OK",
  "error_message": ""
}
Configuration

  • monitoring_svc_end_point (optional. must be valid http|https url) - It is the service endpoint to which we will have to post all the captured metrics.

  • monitoring_enabled (optional. true|false, default value is true) - Enables or disables daemon monitoring. By default it is set to true. When enabled DAemon captures the request and resposne metrics and post to the end point.

Service endpoint

POST http://127.0.0.1/beta/event

Alerts/Notifications

It is for alerting the user in case of any issues/errors/warning from the daemon/service, and also pass on some critical information when needed. Alerts depends on an external webhook or service endpoint o relay the messages to the configured email address.

configuration

  • alerts_email (optional unless metrics enabled. must be a valid email address) - an email for the alerts when there is an issue/warning/error/information to be sent.

  • notification_svc_end_point (optional unless metrics enabled. must be a valid webhook/service end point) - Its service endpoint or a web hook to receive the alerts/notifications and relay it to alert email, which is configured the configuration.

Service endpoint

POST http://127.0.0.1/beta/notify

Sample Payload

{
    "recipient":"abc.comm@gmail.com",
    "message":"From the API",
    "details":"From the API",
    "component":"daemon",
    "component_id":"ad",
    "type":"INFO",
    "level":"10"
}

and the Daemon ID will be added to the request header as Access-Token

req.Header.Set("Access-Token", daemonID)
Configuration in JSON format

This is the sample configuration to enable metrics and heartbeat

  {
      "monitoring_enabled"        : true,
      "monitoring_svc_end_point"  : "http://demo3208027.mockable.io/metrics",
      "notification_svc_end_point": "http://demo3208027.mockable.io/notify",
      "alerts_email"               : "xyz.abc@myorg.io",
      "service_heartbeat_type"    : "http",
      "heartbeat_svc_end_point"   : "http://localhost:25000/heartbeat"  
   }

Documentation

Overview

package for monitoring and reporting the daemon metrics

package for monitoring and reporting the daemon metrics

package for monitoring and reporting the daemon metrics

package for monitoring and reporting the daemon metrics

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConvertStructToJSON 

func ConvertStructToJSON(payLoad interface{}) ([]byte, error)

convert the given struct to its corresponding json.

func GenXid 

func GenXid() string

Generate a unique global Id

func GetDaemonID 

func GetDaemonID() string

generates DaemonID nad returns i.e. DaemonID = HASH (Org Name, Service Name, daemon endpoint)

func GetSize 

func GetSize(v interface{}) uint64

Generic utility to determine the size of the srtuct passed

func GetValue 

func GetValue(md metadata.MD, key string) string

Get the value of the first Pair

func HeartbeatHandler 

func HeartbeatHandler(rw http.ResponseWriter, r *http.Request)

Heartbeat request handler function : upon request it will hit the service for status and wraps the results in daemons heartbeat

func Publish 

func Publish(payload interface{}, serviceUrl string) bool

convert the payload to JSON and publish it to the serviceUrl passed

func PublishRequestStats 

func PublishRequestStats(commonStat *CommonStats, inStream grpc.ServerStream) bool

Create a request Object and Publish this to a service end point

func PublishResponseStats 

func PublishResponseStats(commonStats *CommonStats, duration time.Duration, err error) bool

Publish response received as a payload for reporting /metrics analysis If there is an error in the response received from the service, then send out a notification as well.

func RegisterDaemon 

func RegisterDaemon(serviceURL string) bool

New Daemon registration. Generates the DaemonID and use that as getting access token

func SetDaemonGrpId 

func SetDaemonGrpId(grpId string)

setter method for daemonGroupID

func SetIsNoAlertsConfig 

func SetIsNoAlertsConfig(state bool)

set the no alerts URL and email State

func SetNoHeartbeatURLState 

func SetNoHeartbeatURLState(state bool)

set the no heartbeat URL State

func ValidateHeartbeatConfig 

func ValidateHeartbeatConfig() error

validates the heartbeat configurations

func ValidateNotificationConfig 

func ValidateNotificationConfig() error

validates the heartbeat configurations

Types

type CommonStats 

type CommonStats struct {
	ID                  string
	ServiceMethod       string
	RequestReceivedTime string
	OrganizationID      string
	ServiceID           string
	GroupID             string
	DaemonEndPoint      string
	Version             string
}

func BuildCommonStats 

func BuildCommonStats(receivedTime time.Time, methodName string) *CommonStats

type DaemonHeartbeat 

type DaemonHeartbeat struct {
	DaemonID         string `json:"daemonID"`
	Timestamp        string `json:"timestamp"`
	Status           string `json:"status"`
	ServiceHeartbeat string `json:"serviceheartbeat"`
}

define heartbeat data model. Service Status JSON object Array marshalled to a string

func GetHeartbeat 

func GetHeartbeat(serviceURL string, serviceType string, serviceID string) (heartbeat DaemonHeartbeat, err error)

prepares the heartbeat, which includes calling to underlying service DAemon is serving

func (*DaemonHeartbeat) Check 

func (service *DaemonHeartbeat) Check(ctx context.Context, req *grpc_health_v1.HealthCheckRequest) (*grpc_health_v1.HealthCheckResponse, error)

Check implements `service Health`.

func (*DaemonHeartbeat) Watch 

func (service *DaemonHeartbeat) Watch(*grpc_health_v1.HealthCheckRequest, grpc_health_v1.Health_WatchServer) error

Watch implements `service Watch todo for later`.

type Notification 

type Notification struct {
	DaemonID  string `json:"component_id"`
	Timestamp string `json:"timestamp"`
	Recipient string `json:"recipient"`
	Message   string `json:"message"`
	Details   string `json:"details"`
	Component string `json:"component"`
	Type      string `json:"type"`
	Level     string `json:"level"`
}

define heartbeat data model. Service Status JSON object Array marshalled to a string

func (*Notification) Send 

func (alert *Notification) Send() bool

function for sending an alert to a given endpoint

type RegisterDaemonPayload 

type RegisterDaemonPayload struct {
	NetworkID int    `json:"netId"`
	DaemonID  string `json:"daemonId"`
}

type RequestStats 

type RequestStats struct {
	Type                       string `json:"type"`
	RegistryAddressKey         string `json:"registry_address_key"`
	EthereumJsonRpcEndpointKey string `json:"ethereum_json_rpc_endpoint"`
	RequestID                  string `json:"request_id"`
	InputDataSize              string `json:"input_data_size"`
	ServiceMethod              string `json:"service_method"`
	RequestReceivedTime        string `json:"request_received_time"`
	OrganizationID             string `json:"organization_id"`
	ServiceID                  string `json:"service_id"`
	GroupID                    string `json:"group_id"`
	DaemonEndPoint             string `json:"daemon_end_point"`
	Version                    string `json:"version"`
}

Request stats that will be captured

type Response 

type Response struct {
	ServiceID string `json:"serviceID"`
	Status    string `json:"status"`
}

type ResponseStats 

type ResponseStats struct {
	Type                       string `json:"type"`
	RegistryAddressKey         string `json:"registry_address_key"`
	EthereumJsonRpcEndpointKey string `json:"ethereum_json_rpc_endpoint"`
	RequestID                  string `json:"request_id"`
	OrganizationID             string `json:"organization_id"`
	ServiceID                  string `json:"service_id"`
	GroupID                    string `json:"group_id"`
	ServiceMethod              string `json:"service_method"`
	ResponseSentTime           string `json:"response_sent_time"`
	RequestReceivedTime        string `json:"request_received_time"`
	ResponseTime               string `json:"response_time"`
	ResponseCode               string `json:"response_code"`
	ErrorMessage               string `json:"error_message"`
	Version                    string `json:"version"`
}

Response stats that will be captured and published

type Status 

type Status int

status enum 

const (
	Offline  Status = 0 // Returns if none of the services are online
	Online   Status = 1 // Returns if any of the services is online
	Warning  Status = 2 // if daemon has issues in extracting the service state
	Critical Status = 3 // if the daemon main thread killed or any other critical issues
)

func (Status) String 

func (state Status) String() string

Converts the enum index into enum names

type TokenGenerated 

type TokenGenerated struct {
	Status string `json:"status"`
	Data   struct {
		Token string `json:"token"`
	} `json:"data"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.