In order to make some robustness tests on a multi pod system, I need to simulate somehow a pod failure. I don't want to use scale down on the deployment since it is less realistic. Deleting pods is also not very functional in my case since pods are coming back in a short time.
I have checked the responses on the: How to simulate Power Failure In Kubernetes And don't match my needs. Any suggestion?
You can check this tool called Chaos Mesh.
This has options for Pod Failure, Network Failure, Creating Stress, IO Chaos etc. Chaos mesh is specifically designed for Kubernetes and its open source.
There is also another great tool called Litmus. This is also designed specifically for kubernetes.
To simulate a network issue I would suggest to shutdown the one of the node or mess with it`s IP addresses so that it would become unreachable.
Another way would to simulate max out resources . You could exec into a pod and run something that will consume a lot of resources (CPU, RAM) while having set up limits for them. Alternatively you can set up limit below the actual usage.
You can also set up probes and simulate their failures. Failed Liveness probe would force pod to be restarted.
Lastly, you can also check Chaos Controller. It provides controller for chaos testing and supports irch set of failure scenarios.
Maybe you can go with stressing out memory usage so that it will get killed due to OOM issue. and try to kill main process running inside container that will also restart pod. and The last option I would suggest configure wrong liveness and readiness probe.
