azure-k8s-metrics-adapter

README

Azure Kubernetes Metrics Adapter

An implementation of the Kubernetes Custom Metrics API and External Metrics API for Azure Services.

This adapter enables you to scale your application deployment pods running on AKS using the Horizontal Pod Autoscaler (HPA) with External Metrics from Azure Resources (such as Service Bus Queues) and Custom Metrics stored in Application Insights.

Learn more about using an HPA to autoscale with external and custom metrics.

Checkout a video showing how scaling works with the adapter, deploy the adapter or learn by going through the walkthrough.

This was build using the Custom Metric Adapter Server Boilerplate project.

Project Status: Alpha

Walkthrough

Try out scaling with External Metrics using the a Azure Service Bus Queue in this walkthrough.

Try out scaling with Custom Metrics using Requests per Second and Application Insights in this walkthrough

Deploy

Requires some set up on your AKS Cluster and Metric Server deployed to your cluster.

kubectl apply -f https://raw.githubusercontent.com/Azure/azure-k8s-metrics-adapter/master/deploy/adapter.yaml

After deployment you can create an Horizontal Pod Auto Scaler (HPA) to scale of your external metric of choice:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
 name: consumer-scaler
spec:
 scaleTargetRef:
   apiVersion: extensions/v1beta1
   kind: Deployment
   name: consumer
 minReplicas: 1
 maxReplicas: 10
 metrics:
  - type: External
    external:
      metricName: queuemessages
      metricSelector:
        matchLabels:
          metricName: Messages
          resourceGroup: sb-external-example
          resourceName: sb-external-ns
          resourceProviderNamespace: Microsoft.Servicebus
          resourceType: namespaces
          aggregation: Total
          filter: EntityName_eq_externalq
      targetValue: 30

And your that's it to enable auto scaling on External Metric. Checkout the samples for more examples.

Verifying the deployment

You can also can also query the api to available metrics:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

To Query for a specific custom metric (not currently supported):

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test/pods/*/custom-metric" | jq .

To query for a specific external metric:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/test/queuemessages?labelSelector=resourceProviderNamespace=Microsoft.Servicebus,resourceType=namespaces,aggregation=Total,filter=EntityName_eq_externalq,resourceGroup=sb-external-example,resourceName=sb-external-ns,metricName=Messages" | jq .

External Metrics

Requires k8 1.10+

See a full list of hundreds of available azure external metrics that can be used.

Common external metrics to use for autoscaling are:

• Azure ServiceBus Queue - Message Count - example
• Azure Storage Queue - Message Count
• Azure Eventhubs

Custom Metrics

Custom metrics are currently retrieved from Application Insights. Currently you will need to use an AppId and API key to authenticate and retrieve metrics from Application Insights. View a list of basic metrics that come out of the box and see sample values at the AI api explorer.

note: Metrics that have multi parts currently need will be passed as with a separator of - in place of AI separator of /. An example is performanceCounters/requestsPerSecond should be specified as performanceCounters-requestsPerSecond

Common Custom Metrics are:

• Requests per Second (RPS) - example

Azure Setup

Security

Authenticating with Azure Monitor can be achieved via a variety of authentication mechanisms. (full list)

We recommend to use one of the following options:

• Azure Managed Service Identity (MSI)
• Azure AD Application ID and Secret
• Azure AD Application ID and X.509 Certificate

The Azure AD entity needs to have Monitoring Reader permission on the resource group that will be queried. More information can be found here.

Using Azure Managed Service Identity (MSI)

Enable Managed Service Identity on each of your AKS vms:

There is a known issue when upgrading a AKS cluster with MSI enabled. After the AKS upgrade you will lose your MSI setting and need to re-enable it. An alternative may be to use aad-pod-identity

export RG=<aks resource group> 
export CLUSTER=<aks cluster name> 

NODE_RG="$(az aks show -n $CLUSTER -g $RG | jq -r .nodeResourceGroup)"
az vm list -g $NODE_RG
VMS="$(az vm list -g $NODE_RG | jq -r '.[] | select(.tags.creationSource | . and contains("aks")) | .name')"

while read -r vm; do
    echo "updating vm $vm..."
    msi="$(az vm identity assign -g $NODE_RG -n $vm | jq -r .systemAssignedIdentity)"
done <<< "$VMS"

Give access to the resource the MSI needs to access for each vm:

export RG=<aks resource group> 
export CLUSTER=<aks cluster name> 
export ACCESS_RG=<resource group with metrics>

NODE_RG="$(az aks show -n $CLUSTER -g $RG | jq -r .nodeResourceGroup)"
az vm list -g $NODE_RG
VMS="$(az vm list -g $NODE_RG | jq -r '.[] | select(.tags.creationSource | . and contains("aks")) | .name')"

while read -r vm; do
    echo "getting vm identity $vm..."
    msi="$(az vm identity show -g $NODE_RG -n $vm | jq -r .principalId)"

    echo "adding access with msi $msi..."
    az role assignment create --role "Monitoring Reader" --assignee-object-id $msi --resource-group $ACCESS_RG
done <<< "$VMS"

Using Azure AD Application ID and Secret

See how to create an example deployment.

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CLIENT_SECRET: Specifies the app secret to use.

Azure AD Application ID and X.509 Certificate

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CERTIFICATE_PATH: Specifies the certificate Path to use.
• AZURE_CERTIFICATE_PASSWORD: Specifies the certificate password to use.

Subscription Information

The use the adapter your Azure Subscription must be provided. There are a few ways to provide this information:

• Azure Instance Metadata - If you are running the adapter on a VM in Azure (for instance in an AKS cluster) there is nothing you need to do. The Subscription Id will be automatically picked up from the Azure Instance Metadata endpoint
• Environment Variable - If you are outside of Azure or want full control of the subscription that is used you can set the Environment variable SUBSCRIPTION_ID on the adapter deployment. This takes precedence over the Azure Instance Metadata.
• On each HPA - you can work with multiple subscriptions by supplying the metric selector subscriptionID on each HPA. This overrides Environment variables and Azure Instance Metadata settings.

Contributing

See Contributing for more information.

Issues

Report any issues in the Github issues.

Roadmap

See the Projects tab for current roadmap.

Reporting Security Issues

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at secure@microsoft.com. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.