Context:
I currently have a Python Flask app which when receives a request, it automatically creates a Kubernetes job to run a process. Once the job finishes, it posts back the results to a specific endpoint (never changes). Depending on the specifics of the request, this process can vary greatly in length (time) and in resource requirements. The reason I'm using Jobs for this is due to their ability to restart on failure.
Problem:
Even with allowing maximum restarts, there will be times where the process will error. I want to implement something along the lines of:
- If the job finishes successfully, do nothing.
- If the job fails after reaching max retries, then run
foo()
foo() is a very simple Python function that will send an HTTP request with the parameters of the request that resulted in the job not successfully completing. It does not have to be a Python function, but it is preferred.
Note: I wrap everything in a broad error exception within the python function that the job needs to perform, however, there might be times that the job will fail due to insufficient memory.
