kubernetes-traffic-manager

module
v0.0.0-...-ad5601e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 10, 2019 License: Apache-2.0

README

Introduction

kubernetes-traffic-manager is a traffic management tools for kubernetes, it implements envoy (https://www.envoyproxy.io/) control plane data api to support:

  • Ingress gateway
  • Weighted load balancing
  • Tracing
  • Fault Injection
  • Circuit breaker
  • Runtime metrics

Installation

helm install --name kubernetes-traffic-manager helm/kubernetes-traffic-manager

or install with kubernetes-traffic-monitor

helm install --set monitor.enabled=true --name kubernetes-traffic-manager helm/kubernetes-traffic-manager

Note that for helm 3.0, --name is not needed

Ingress gateway

# deploy sample application
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml

# deploy k8s ingress resource
kubectl apply -f samples/ingress.yaml

# wait awhile then run
INGRESS_HOST=`kubectl get svc traffic-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`
echo $INGRESS_HOST
# if it is empty, try 
INGRESS_HOST=`kubectl get svc traffic-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'`

#access productpage service
curl ${INGRESS_HOST}/productpage
#access review service
curl -v ${INGRESS_HOST}/reviews/0
#access promethues service
curl ${INGRESS_HOST}/api/v1/label/__name__/values

Ingress gateway with TLS

This example will use certificate generated by Let's Encrypt.

Ensure your traffic-ingress service has a public loadbalancer ip(${INGRESS_HOST}), and you need to apply a hostname for the ip, when the host is ready:

mkdir certbot
docker run -it -v ${PWD}/certbot:/etc/letsencrypt certbot/certbot certonly --manual  --preferred-challenges http -d (your host name)

The command will show message like following:

Create a file containing just this data:

jLpYJvXE4mP32AgP42O4Ws-iT7_Z9St2pOjdlbqhkhA.3Jf5jGBx3a6iQm1qUIJUlPjc7UrWMNhsTCzFHOV5FgM

And make it available on your web server at this URL:

http://(your host name)/.well-known/acme-challenge/jLpYJvXE4mP32AgP42O4Ws-iT7_Z9St2pOjdlbqhkhA

Open another terminal, change value of RESPONSE_BODY in samples/http-text-response.yaml to the displayed data(in above example, jLpYJvXE4mP32AgP42O4Ws...).

kubctl apply -f samples/http-text-response.yaml
cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: https-ingress
spec:
  rules:
  - host: (your host name)
    http:
      paths:
      - path: /.well-known/acme-challenge/jLpYJvXE4mP32AgP42O4Ws-iT7_Z9St2pOjdlbqhkhA
        backend:
          serviceName: http-text-response
          servicePort: 8080
EOF

Note that you need to change host and path value to the value in your cerbot output.

After running above commands, switch to cerbot terminal, press ENTER to let command conntinue, cerbot will generate the tls secrets to file. Then run follwing command:

kubectl create secret tls ingressgateway-certs   --key certbot/live/(your host name)/privkey.pem --cert certbot/live/(your host name)/fullchain.pem

cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: https-ingress
spec:
 tls:
 - hosts:
   - (your host name)
   secretName: ingressgateway-certs
 rules:
 - host: (your host name)
   http:
     paths:
     - path: /productpage
       backend:
         serviceName: productpage
         servicePort: 9080
EOF

Now open browser and browse https://(your host name)/productpage

Runtime metrics

# generate traffic
curl -v ${INGRESS_HOST}/reviews/0

# query runtime metrics
curl ${INGRESS_HOST}/api/v1/label/__name__/values |jq
curl -G http://${INGRESS_HOST}/api/v1/query --data-urlencode "query=envoy_cluster_outbound_upstream_rq_completed{instance='(Ingress PodIP):8900', envoy_cluster_name='9080|default|reviews'}"|jq

Load Balancing

Resource Labels Default Description
Service traffic.lb.policy ROUND_ROBIN load balance policy: ROUND_ROBIN, LEAST_REQUEST, RING_HASH, RANDOM, MAGLEV
Service traffic.hash.cookie.name "" cookie hash policy
Service traffic.hash.cookie.ttl 0 generate cookie with ttl (seconds)
Service traffic.hash.header.name "" http header name for hash policy
Pod, Deployment, StatefulSet, DaemonSet traffic.endpoint.weight 100 weight value for related pods [0-128] 
# Default lb policy is ROUND_ROBIN
kubectl label deployment reviews-v1 traffic.endpoint.weight=100 
kubectl label deployment reviews-v2 traffic.endpoint.weight=10 
kubectl label deployment reviews-v3 traffic.endpoint.weight=0

# The value should be 2, since reviews-v3 has weight 0
curl -G http://${INGRESS_HOST}/api/v1/query --data-urlencode "query=envoy_cluster_outbound_membership_total{envoy_cluster_name='9080|default|reviews'}"|jq

# Use cookie hash policy
kubectl label svc reviews traffic.lb.policy=RING_HASH
kubectl label svc reviews traffic.hash.cookie.name="mycookie"
kubectl label svc reviews traffic.hash.cookie.ttl="100000"

kubectl label deployment reviews-v3 traffic.endpoint.weight=20

curl -v  http://${INGRESS_HOST}/reviews/0
# The http response should contains set-cookie, for example:
# set-cookie: mycookie="3acd918773ba09c5"; Max-Age=100; HttpOnly

#following request should always send to same review pod, it should always contain "ratings" or always be without "ratings"
curl -v  -H "Cookie: mycookie=3acd918773ba09c5" http://${INGRESS_HOST}/reviews/0

kubectl delete -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml

Reference:

Enable envoy

When user label a pod or deployment with "traffic.envoy.enabled=true", the related pods' traffic will be managed. Runtime metrics and load balancing will be applied like traffic from ingress pod.

By default, all envoy enabled pods' outcoming traffic will be blocked. You need to add traffic.port.(port number)=(protocol) labels for service or pod to unblock traffic on certain port. The protocol can be http or tcp or direct(bypass envoy loadbalacing)

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml

#unblock reviews service traffic
kubectl label svc reviews traffic.port.9080=http

#enable envoy for zipkin pod
kubectl label deployment traffic-zipkin traffic.envoy.enabled=true

#lb policy configuration on reviews sevice should be applied
kubectl exec traffic-zipkin-694c7884d5-bqdvm -- curl http://reviews:9080/reviews/0

# runtime metrics
curl -G http://${INGRESS_HOST}/api/v1/query --data-urlencode "query=envoy_cluster_outbound_upstream_rq_completed{instance='(traffic-zipkin PodIP):8900', envoy_cluster_name='9080|default|reviews'}"|jq

service in ingress configuration will be automatically annotated(not labeled) with http protocol, so no need to unblock them.

Fault Injection

Resource Labels Default Description
Pod, Service traffic.fault.delay.time 0 delay time in nanoseconds
Pod, Service traffic.fault.delay.percentage 0 percentage of requests to be delayed for time
Pod, Service traffic.fault.abort.status 0 abort with http status
Pod, Service traffic.fault.abort.percentage 0 percentage of requests to be aborted
Pod, Service traffic.rate.limit 0 rate limit number in Kbps on each client

Ingress does not support Fault Injection

kubectl label svc reviews traffic.fault.delay.time=3000
kubectl label svc reviews traffic.fault.delay.percentage=100

# should delay 3 seconds
kubectl exec traffic-zipkin-694c7884d5-bqdvm -- curl http://reviews:9080/reviews/0

kubectl label svc reviews traffic.fault.abort.status=505
kubectl label svc reviews traffic.fault.abort.percentage=100

# should delay 3 seconds and return http 505
kubectl exec traffic-zipkin-694c7884d5-bqdvm -- curl -v http://reviews:9080/reviews/0

kubectl label svc reviews traffic.fault.abort.percentage-
kubectl label svc reviews traffic.fault.abort.status-
kubectl label svc reviews traffic.fault.delay.percentage-
kubectl label svc reviews traffic.fault.delay.time-
  
 # should return normal
kubectl exec traffic-zipkin-694c7884d5-bqdvm -- curl -v http://reviews:9080/reviews/0

kubectl delete -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml

Tracing

Resource Labels Default Description
Pod, Service traffic.tracing.enabled false enable tracing for requests to or from envoy enabled pods of this service
Pod, Service traffic.tracing.sampling 100 percentage of tracing sampling (float) 
kubectl label deployment traffic-zipkin traffic.envoy.enabled=false --overwrite

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml

kubectl label svc productpage details ratings reviews traffic.port.9080=http

kubectl label deployment productpage-v1 details-v1 ratings-v1 reviews-v1 reviews-v2 reviews-v3 traffic.envoy.enabled=true

kubectl label svc productpage details ratings reviews traffic.tracing.enabled=true
# sampling rate 50%
kubectl label svc productpage details ratings reviews traffic.tracing.sampling=50

#repeat following command 10 times
kubectl exec traffic-zipkin-694c7884d5-bqdvm -- curl -v http://productpage:9080:/productpage

# show traceId, number of returned traceId should be around 5
kubectl exec traffic-zipkin-694c7884d5-x4pn9 curl http://localhost:9411/api/v2/traces?limit=100 | python3 -c "import sys, json; print([x[0]['traceId'] for x in json.load(sys.stdin)])"

Like Istio, Applications need to propagate appropriate HTTP headers, reference: https://istio.io/docs/tasks/telemetry/distributed-tracing/overview/

Check running envoy proxy instances on each node

Find envoy manager on target node
kubectl get pod -o wide
NAME                              READY     STATUS    RESTARTS   AGE       IP             NODE
traffic-envoy-manager-6f7nw       1/1       Running   0          9m        (node ip)  (targert node name)
Check running envoy proxy instances
shell> kubectl exec traffic-envoy-manager-6f7nw ./envoy-tools
ID                                                                 |Pod                            |Namespace   |Status                  |State     |
--                                                                 |---                            |---------   |------                  |-----     |
86e56e0dc8e6a734fdb547aab60d9720117231742a32bdeaabc7dea6316a2f7b   |reviews-v3-5df889bcff-xdrwb    |default     |Up Less than a second   |running   |
a55a9126aef6f402e71c6c9ee61c3c0674aef8b71aeb40334fc1154695d80410   |reviews-v2-7ff5966b99-krw9s    |default     |Up 1 second             |running   |

shell> kubectl exec traffic-envoy-manager-6f7nw -- ./envoy-tools -id (prefix of the envoy id) -log

Check envoy configuration

kubectl exec traffic-control-89778f5d8-nmvrn -- ./envoy-config --nodeId reviews-v3-5df889bcff-f2hgh.default

node id is pod name and pod namespace, ingress node id is traffic-ingress.

Circuit Breaker

Resource Labels Default Description
Service traffic.retries.max 0 max retries number
Service traffic.connection.max 0 max number of connection
Service traffic.request.max-pending 0 max pending requests
Service traffic.request.max 0 max requests

Other Configuration Labels

Resource Labels Default Description
Pod, Deployment, StatefulSet, DaemonSet traffic.envoy.enabled false whether to enable envoy docker for related pods
Pod, Service traffic.port.(port number) None protocol for the port on service (http, tcp, direct)
Pod, Service traffic.request.timeout 0 timeout in nanoseconds
Pod, Service traffic.retries.5xx 0 number of retries for 5xx error
Pod, Service traffic.retries.connect-failure 0 number of retries for connect failure
Pod, Service traffic.retries.gateway-error 0 number of retries for gateway error
Service traffic.connection.timeout 60 * 1e9 timeout in nanoseconds

Note that all the service label configuration requires client pod's envoy enabled.

Components

envoy-manager

Responsible to start/stop envoy container for traffic-control

When a pod is envoy enabled, envoy-manager will start a envoy docker instance:

  • Envoy docker git repo: https://github.com/luguoxiang/traffic-envoy
  • The docker instance will share corresponding pod's network(docker container network mode).
  • The iptable config of the pod network will be changed to redirect all outcoming traffic to envoy container listen port.
  • The pod will be annotated with traffic.envoy.proxy=(docker id)

When user label the service with "traffic.envoy.enabled=false"

  • the docker instance will be deleted
  • the iptable config will be restored
  • the pod will be annotated with traffic.envoy.proxy=""

traffic-control

traffic-control is a control plane implementation of envoy proxy (https://www.envoyproxy.io/).

Directories

Path Synopsis
cmd
control-plane
create-secret
envoy-config
envoy-manager
envoy-tools
pkg
annotation
client
docker
envoy
envoy/cluster
envoy/common
envoy/endpoint
envoy/listener
envoy/listener/ingress
kubernetes

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.