azure-k8s-metrics-adapter

README

Azure Kubernetes Metrics Adapter

An implementation of the Kubernetes Custom Metrics API and External Metrics API for Azure Services.

This adapter enables you to scale your application deployment pods running on AKS using the Horizontal Pod Autoscaler (HPA) with External Metrics from Azure Resources (such as Service Bus Queues) and Custom Metrics stored in Application Insights.

Try it out:

• External metric scaling video
• Custom metric scaling video
• Deploy the adapter
• Azure Service Bus queue walkthrough
• Request Per Second (RPS) walkthrough

This was build using the Custom Metric Adapter Server Boilerplate project. Learn more about using an HPA to autoscale with external and custom metrics.

Project Status: Alpha

Walkthrough

Try out scaling with External Metrics using the a Azure Service Bus Queue in this walkthrough.

Try out scaling with Custom Metrics using Requests per Second and Application Insights in this walkthrough

Quick-Start Deploy

This describes the basic steps for deploying the metric adapter. For full deployment details see how to set up on your AKS Cluster and checkout the samples. Make sure the Metric Server is already deployed to your cluster.

Create a Service Principle and Secret:

az ad sp create-for-rbac -n "azure-k8s-metric-adapter-sp" --role "Monitoring Reader" --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}

#use values from service principle created above to create secret
kubectl create secret generic azure-k8s-metrics-adapter -n custom-metrics \
  --from-literal=azure-tenant-id=<tenantid> \
  --from-literal=azure-client-id=<clientid>  \
  --from-literal=azure-client-secret=<secret>

Deploy the adapter:

kubectl apply -f https://raw.githubusercontent.com/Azure/azure-k8s-metrics-adapter/master/deploy/adapter.yaml

Deploy a metric configuration (requires you to configure the file below with your settings to a Service Bus Queue):

kubectl apply -f https://raw.githubusercontent.com/Azure/azure-k8s-metrics-adapter/master/samples/resources/externalmetric-example.yaml

There is also a Helm chart available for deployment for those using Helm in their cluster.

Deploy a Horizontal Pod Auto Scaler (HPA) to scale of your external metric of choice:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
 name: consumer-scaler
spec:
 scaleTargetRef:
   apiVersion: extensions/v1beta1
   kind: Deployment
   name: consumer
 minReplicas: 1
 maxReplicas: 10
 metrics:
  - type: External
    external:
      metricName: queuemessages
      targetValue: 30

Checkout the samples for more examples and details.

Verifying the deployment

You can also can query the api to verify it is installed:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

To Query for a specific custom metric:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test/pods/*/custom-metric" | jq .

To query for a specific external metric:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/test/queuemessages" | jq .

External Metrics

Requires k8s 1.10+

See a full list of hundreds of available azure external metrics that can be used.

Common external metrics to use for autoscaling are:

• Azure ServiceBus Queue - Message Count - example

Custom Metrics

Custom metrics are currently retrieved from Application Insights. View a list of basic metrics that come out of the box and see sample values at the AI api explorer.

Common Custom Metrics are:

• Requests per Second (RPS) - example

Azure Setup

Security

Authenticating with Azure Monitor can be achieved via a variety of authentication mechanisms. (full list)

Use one of the following options:

• Azure AD Pod Identity (aad-pod-identity)
• Azure AD Application ID and Secret
• Azure AD Application ID and X.509 Certificate

The Azure AD entity needs to have Monitoring Reader permission on the resource group that will be queried. More information can be found here.

Using Azure AD Pod Identity

aad-pod-identity is currently in beta and allows to bind a user managed identity or a service principal to a pod. That means that instead of using the same managed identity for all the pod running on a node like explained above, you are able to get a specific identity with specific RBAC for a specific pod.

Using this project requires to deploy a bit of infrastructure first. You can do it following the Get started page of the project.

Once the aad-pod-identity infrastructure is running, you need to create an Azure identity scoped to the resource group you are monitoring:

az identity create -g {ResourceGroup1} -n custom-metrics-identity

Assign Monitoring Reader to it

az role assignment create --role "Monitoring Reader" --assignee <principalId> --scope /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}

Note: you need to assign this role to all resources groups that you want the identity to be able to read Azure Monitor data.

As documented here aad-pod-identity uses the service principal of your Kubernetes cluster to access the Azure resources. You need to give this service principal the rights to use the managed identity created before:

az role assignment create --role "Managed Identity Operator" --assignee <servicePrincipalId> --scope /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-metrics-identity

Install the Azure Identity to your Kubernetes cluster:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: custom-metrics-identity
spec:
  type: 0
  ResourceID: /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-metrics-identity
  ClientID: <clientid>

Install the Azure Identity Binding on your Kubernetes cluster:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: custom-metrics-identity-binding
spec:
  AzureIdentity: custom-metrics-identity
  Selector: custom-metrics-identity

Note: pay attention to the name of the selector above. You will need to use it to bind the identity to your pod.

If you use the Helm Chart to deploy the custom metrics adapter to your Kubernetes cluster, you can configure Azure AD Pod Identity directly in the config values:

azureAuthentication:
  method: aadPodIdentity
  # if you use aadPodIdentity authentication
  azureIdentityName: "custom-metrics-identity"
  azureIdentityBindingName: "custom-metrics-identity-binding"
  # The full Azure resource id of the managed identity (/subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{IdentityName})
  azureIdentityResourceId: ""
  # The Client Id of the managed identity
  azureIdentityClientId: ""

Switch method to aadPodIdentity a give the value for the Azure Identity resource id and client id, for example:

helm install ./charts/azure-k8s-metrics-adapter --set azureAuthentication.method="aadPodIdentity" \
  --set azureAuthentication.azureIdentityResourceId="/subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{IdentityName}" \
  --set azureAuthentication.azureIdentityClientId="{ClientId}" \
  --name "custom-metrics-adapter"

Using Azure AD Application ID and Secret

See how to create an example deployment.

Create a service principal scoped to the resource group the resource you monitoring and assign Monitoring Reader to it:

az ad sp create-for-rbac -n "adapter-sp" --role "Monitoring Reader" --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CLIENT_SECRET: Specifies the app secret to use.

Deploy the environment variables via secret:

 kubectl create secret generic adapter-service-principal -n custom-metrics \
  --from-literal=azure-tenant-id=<tenantid> \
  --from-literal=azure-client-id=<clientid>  \
  --from-literal=azure-client-secret=<secret>

Azure AD Application ID and X.509 Certificate

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CERTIFICATE_PATH: Specifies the certificate Path to use.
• AZURE_CERTIFICATE_PASSWORD: Specifies the certificate password to use.

Subscription Information

The use the adapter your Azure Subscription must be provided. There are a few ways to provide this information:

• Azure Instance Metadata - If you are running the adapter on a VM in Azure (for instance in an AKS cluster) there is nothing you need to do. The Subscription Id will be automatically picked up from the Azure Instance Metadata endpoint
• Environment Variable - If you are outside of Azure or want full control of the subscription that is used you can set the Environment variable SUBSCRIPTION_ID on the adapter deployment. This takes precedence over the Azure Instance Metadata.
• On each HPA - you can work with multiple subscriptions by supplying the metric selector subscriptionID on each HPA. This overrides Environment variables and Azure Instance Metadata settings.

FAQ

• Can I scale with Azure Storage queues?
  • Not currently. The Azure Storage Queue only reports it's capacity metrics daily. Follow this issue for updates.
• The metrics numbers look slightly off compared to portal or Service Bus Explorer. Why are the values not exact?
  • Azure Monitor has a delay (30s - 2 mins) in reported values. This delay can also be seen in the Azure Monitor dashboard in the portal. There is also a delay in the values reported when using Application Insights.

Contributing

See Contributing for more information.

Issues

Report any issues in the Github issues.

Roadmap

See the Projects tab for current roadmap.

Reporting Security Issues

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at secure@microsoft.com. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.