pod-restarter-go

command module
v0.0.0-...-a1b7dae Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 10, 2023 License: Apache-2.0 Imports: 8 Imported by: 0

README

pod-restarter: restarts Pods in a failed state using client-go kubernetes packages

test

Performs the following steps:

  • Looks for latest Pod Events that matches an Event Reason and Message
  • If there are matching Pods, these Pods will go through a sequence of steps before they get deleted:
    • verify Pod exists
    • verify Pod has owner/controller
    • verify Pod has not been scheduled to be deleted
    • verify Pod is in a Failing State (Pending/Failed or Running with failing containers)
  • If all above checks pass, Pod will be deleted

These steps are repeated in a loop on a polling interval basis.

Configuring pod-restarter

pod-restarter is configurable through cli parameters.

--polling-interval
  • Delete Pods that have matching Events with default Reason and Message every poll interval (seconds).
  • Default value: 30 (seconds)
./pod-restarter --polling-interval 10
--dry-run
  • Logs pod-restarter actions but don't actually delete any pods.
  • Default value: disabled
./pod-restarter --dry-run
--reason and --error-message
  • These parameters work together because every Event has a Reason and a related Message.
  • These parameters are used for identifying failing Pods that match Event Reason and Message.
  • Default values:
    • "FailedCreatePodSandBox" (Event Reason)
    • "container veth name provided (eth0) already exists" (Event Message)
# delete Pods that have Events with default Reason "FailedCreatePodSandBox" and default Message "container veth name provided (eth0) already exists"
./pod-restarter

# delete Pods that have Events with Reason "BackOff" and Message "Back-off pulling image"
./pod-restarter --reason "BackOff" --error-message "Back-off pulling image"
--namespace
  • The kubernetes namespavce where pod-restarter should look for Failing Pods.
  • Default value: "" (look for all namespaces)
# delete Pods in namespace default
./pod-restarter --namespace default
--kubeconfig
  • When run locally (outside of cluster), specifies the kubeconfig config.
  • Default value: ~/.kube/config

Run and test on local machine/laptop

# install dependencies
go mod init
go mod tidy

# compile source code into executable binary
go build -o pod-restarter

# run in dry mode
./pod-restarter --dry-run --polling-interval 10

# run unit tests
go test -v ./...

Deploy to k8s with helm

# build container image
task build

# deploy helm chart
task install

# verify helm release
helm list -A

# uninstall helm chart
task uninstall

Simulate Failing Pods and test

# generate failing pods due to pulling wrong images
cd infra/tests
bash generate_pending_pods.sh


# check app logs
kubectl logs -l app=pod-restarter -f

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL