azure-k8s-metrics-adapter

README

Azure Kubernetes Metrics Adapter

An implementation of the Kubernetes Custom Metrics API and External Metrics API for Azure Services.

This adapter enables you to scale your application deployment pods running on AKS using the Horizontal Pod Autoscaler (HPA) with metrics from Azure Resources (such as Service Bus Queues) and custom metrics stored in Application Insights. Learn more about using an HPA to autoscale with with external and custom metrics.

Checkout a video showing how scaling works with the adapter, deploy the adapter or learn by going through the walkthrough.

This was build using the Custom Metric Adapter Server Boilerplate project.

External Metrics

Requires k8 1.10+

See a full list of hundreds of available azure external metrics that can be used.

Common external metrics to use for autoscaling are:

• Azure ServiceBus Queue - Message Count - example
• Azure Storage Queue - Message Count
• Azure Eventhubs

Custom Metrics

Custom Metrics are not currently implemented.

Walkthrough

Check out this walkthrough to try it out.

Deploy

Requires some set up on your AKS Cluster and Metric Server deployed to your cluster.

kubectl apply -f https://raw.githubusercontent.com/jsturtevant/azure-k8-metrics-adapter/master/deploy/adapter.yaml

After deployment you can query the api to avaliable metrics:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .

To Query for a specific custom metric (not currently supported):

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test/pods/*/custom-metric" | jq .

To query for a specific external metric:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/test/queuemessages?labelSelector=resourceProviderNamespace=Microsoft.Servicebus,resourceType=namespaces,aggregation=Total,filter=EntityName_eq_externalq,resourceGroup=sb-external-example,resourceName=sb-external-ns,metricName=Messages" | jq .

Azure Setup

Security

Authenticating with Azure Monitor can be achieved via a variety of authentication mechanisms. (full list)

We recommend to use one of the following options:

• Azure Managed Service Identity (MSI)
• Azure AD Application ID and Secret
• Azure AD Application ID and X.509 Certificate

The Azure AD entity needs to have Monitoring Reader permission on the resource group that will be queried. More information can be found here.

Using Azure Managed Service Identity (MSI)

Enable Managed Service Identity on each of your AKS vms:

There is a known issue when upgrading a AKS cluster with MSI enabled. After the AKS upgrade you will lose your MSI setting and need to re-enable it.

export RG=<aks resource group> 
export CLUSTER=<aks cluster name> 

NODE_RG="$(az aks show -n $CLUSTER -g $RG | jq -r .nodeResourceGroup)"
az vm list -g $NODE_RG
VMS="$(az vm list -g $NODE_RG | jq -r '.[] | select(.tags.creationSource | . and contains("aks")) | .name')"

while read -r vm; do
    echo "updating vm $vm..."
    msi="$(az vm identity assign -g $NODE_RG -n $vm | jq -r .systemAssignedIdentity)"
done <<< "$VMS"

Give access to the resource the MSI needs to access for each vm:

export RG=<aks resource group> 
export CLUSTER=<aks cluster name> 
export ACCESS_RG=<resource group with metrics>

NODE_RG="$(az aks show -n $CLUSTER -g $RG | jq -r .nodeResourceGroup)"
az vm list -g $NODE_RG
VMS="$(az vm list -g $NODE_RG | jq -r '.[] | select(.tags.creationSource | . and contains("aks")) | .name')"

while read -r vm; do
    echo "getting vm identity $vm..."
    msi="$(az vm identity show -g $NODE_RG -n $vm | jq -r .principalId)"

    echo "adding access with msi $msi..."
    az role assignment create --role Reader --assignee-object-id $msi --resource-group $ACCESS_RG
done <<< "$VMS"

Using Azure AD Application ID and Secret

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CLIENT_SECRET: Specifies the app secret to use.

Azure AD Application ID and X.509 Certificate

Required environment variables:

• AZURE_TENANT_ID: Specifies the Tenant to which to authenticate.
• AZURE_CLIENT_ID: Specifies the app client ID to use.
• AZURE_CERTIFICATE_PATH: Specifies the certificate Path to use.
• AZURE_CERTIFICATE_PASSWORD: Specifies the certificate password to use.

Development

Get the source

go get github.com/jsturtevant/azure-k8-metrics-adapter
cd $GOPATH/github.com/jsturtevant/azure-k8-metrics-adapter

Use Skaffold

Before you run the command below be sure to:

• Download skaffold
• Log in to your container registry: docker login
• Have your K8 context set to the cluster you want to deploy to: kubectl config use-context

make dev