0

I'm running dramatiq (an async job executor) in kubernetes. For some reason this process keeps getting killed a few seconds after starting. I'm fairly new to Kubernetes and can't figure out on how to access system logs or logs other than the logs the pod itself outputs.

The only info I'm able to retrieve is the following: [MainThread] [dramatiq.MainProcess] [CRITICAL] Worker with PID 43 exited unexpectedly (code -9). Shutting down...

I'm at a bit of a loss here. Memory uses age and memory config do not appear to be the issue as the pod seems to crash far before the limit is reached. Does anyone have any suggestions on how to maybe get more info regarding the reason of the SIGKILL? I can't seem to reproduce the issue locally.

David Maze
  • 130,717
  • 29
  • 175
  • 215
KPNT
  • 453
  • 3
  • 6
  • 20
  • 1
    Check events `kubectl get events` right after the pod was kill. – Chris Jul 10 '23 at 07:41
  • @Chris thanks, at least this let me know it's because the health probe fails because its connection is refused. – KPNT Jul 10 '23 at 08:48
  • Can you go through this [github issue](https://github.com/Bogdanp/dramatiq/issues/468) once, in this issue the workers are getting restarted or stopped because of OOM kill. Even though you have enough memory or you find the pod are having less memory utilization there is configuration parameter in dramatiq called `mem_limit` try commenting this or assign a huge value and check whether your pods are still getting SIGKILLed. – Kranthiveer Dontineni Jul 10 '23 at 09:52

0 Answers0