chaoskube
![Go Report Card](https://goreportcard.com/badge/github.com/linki/chaoskube)
chaoskube
periodically kills random pods in your Kubernetes cluster.
Why
Test how your system behaves under arbitrary pod failures.
Example
Running it will kill a pod in any namespace every 10 minutes by default.
$ chaoskube
...
INFO[0000] Targeting cluster at https://kube.you.me
INFO[0001] Killing pod kube-system/kube-dns-v20-6ikos
INFO[0601] Killing pod chaoskube/nginx-701339712-u4fr3
INFO[1201] Killing pod kube-system/kube-proxy-gke-earthcoin-pool-3-5ee87f80-n72s
INFO[1802] Killing pod chaoskube/nginx-701339712-bfh2y
INFO[2402] Killing pod kube-system/heapster-v1.2.0-1107848163-bhtcw
INFO[3003] Killing pod kube-system/l7-default-backend-v1.0-o2hc9
INFO[3603] Killing pod kube-system/heapster-v1.2.0-1107848163-jlfcd
INFO[4203] Killing pod chaoskube/nginx-701339712-bfh2y
INFO[4804] Killing pod chaoskube/nginx-701339712-51nt8
...
chaoskube
allows to filter target pods by namespaces, labels and annotations.
See below for details.
How
Get chaoskube
via go get, make sure your current context points to your target cluster and use the --deploy
flag.
$ go get -u github.com/linki/chaoskube
$ chaoskube --deploy
INFO[0000] Dry run enabled. I won't kill anything. Use --no-dry-run when you're ready.
INFO[0000] Targeting cluster at https://kube.you.me
INFO[0000] Deployed quay.io/linki/chaoskube:v0.5.0
By default chaoskube
will be friendly and not kill anything. When you validated your target cluster you may disable dry-run mode. You can also specify a more aggressive interval and other supported flags for your deployment.
$ chaoskube --interval=1m --no-dry-run --debug --deploy
DEBU[0000] Using current context from kubeconfig at /Users/you/.kube/config.
INFO[0000] Targeting cluster at https://kube.you.me
DEBU[0000] Deploying quay.io/linki/chaoskube:v0.5.0
INFO[0000] Deployed quay.io/linki/chaoskube:v0.5.0
Otherwise use the following equivalent manifest file or let it serve as an inspiration.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: chaoskube
spec:
replicas: 1
template:
metadata:
labels:
app: chaoskube
spec:
containers:
- name: chaoskube
image: quay.io/linki/chaoskube:v0.5.0
args:
- --in-cluster
- --interval=1m
- --no-dry-run
- --debug
If you're running in a Kubernetes cluster and want to target the same cluster use the --in-cluster
flag as shown.
If you want to target a different cluster or want to run it locally provide a valid kubeconfig via --kubeconfig
and drop the --in-cluster
flag. By default, it uses your standard kubeconfig path in your home. Whatever is the current context in there will be targeted.
If you want to increase or decrease the amount of chaos change the interval between killings with the --interval
flag. Alternatively, you can increase the number of replicas of your chaoskube
deployment.
Remember that chaoskube
by default kills any pod in all your namespaces, including system pods and itself.
Filtering targets
However, you can limit the search space of chaoskube
by providing label, annotation and namespace selectors.
$ chaoskube --labels 'app=mate,chaos,stage!=production'
...
INFO[0000] Filtering pods by labels: app=mate,chaos,stage!=production
This selects all pods that have the label app
set to mate
, the label chaos
set to anything and the label stage
not set to production
or unset.
You can filter target pods by namespace selector as well.
$ chaoskube --namespaces 'default,testing,staging'
...
INFO[0000] Filtering pods by namespaces: default,staging,testing
This will filter for pods in the three namespaces default
, staging
and testing
.
You can also exclude namespaces and mix and match with the label and annotation selectors.
$ chaoskube \
--labels 'app=mate,chaos,stage!=production' \
--annotations '!scheduler.alpha.kubernetes.io/critical-pod' \
--namespaces '!kube-system,!production'
...
INFO[0000] Filtering pods by labels: app=mate,chaos,stage!=production
INFO[0000] Filtering pods by annotations: !scheduler.alpha.kubernetes.io/critical-pod
INFO[0000] Filtering pods by namespaces: !kube-system,!production
This further limits the search space of the above label selector by also excluding any pods in the kube-system
and production
namespaces as well as ignore all pods that are marked as critical.
The annotation selector can also be used to run chaoskube
as a cluster addon and allow pods to opt-in to being terminated as you see fit. For example, you could run chaoskube
like this:
$ chaoskube --annotations 'chaos.alpha.kubernetes.io/enabled=true'
...
INFO[0000] Filtering pods by annotations: chaos.alpha.kubernetes.io/enabled=true
INFO[0000] No victim could be found. If that's surprising double-check your selectors.
Unless you already use that annotation somewhere, this will initially ignore all of your pods. You could then selectively opt-in individual deployments to chaos mode by annotating their pods with chaos.alpha.kubernetes.io/enabled=true
.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
template:
metadata:
annotations:
chaos.alpha.kubernetes.io/enabled: "true"
spec:
...
Contributing
Feel free to create issues or submit pull requests.