Uvicorn server shutting down unexpectedly

Question

I'm working with FastAPI framework, served by Uvicorn server. My application should run some time consuming numerical computation at a given endpoint (/run). For this I am using 'background_task' from fastAPI (which is basically 'background_task' from Starlette).

When running the application, after some times of nominal behaviour, the server is shut down for some reason.

The logs from the application look like this:

INFO: Started server process [922]
INFO: Waiting for application startup.
DEBUG: None - ASGI [1] Started
DEBUG: None - ASGI [1] Sent {'type': 'lifespan.startup'}
DEBUG: None - ASGI [1] Received {'type': 'lifespan.startup.complete'}
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
DEBUG: ('10.0.2.111', 57396) - Connected
DEBUG: ('10.0.2.111', 57397) - Connected
DEBUG: ('10.0.2.111', 57396) - ASGI [2] Started
DEBUG: ('10.0.2.111', 57396) - ASGI [2] Received {'type': 'http.response.start', 'status': 200, 'headers': '<...>'}
INFO: ('10.0.2.111', 57396) - "GET /run HTTP/1.1" 200
DEBUG: ('10.0.2.111', 57396) - ASGI [2] Received {'type': 'http.response.body', 'body': '<32 bytes>'}
DEBUG: ('10.0.2.111', 57396) - ASGI [3] Started
DEBUG: ('10.0.2.111', 57396) - ASGI [3] Received {'type': 'http.response.start', 'status': 404, 'headers': '<...>'}
INFO: ('10.0.2.111', 57396) - "GET /favicon.ico HTTP/1.1" 404
DEBUG: ('10.0.2.111', 57396) - ASGI [3] Received {'type': 'http.response.body', 'body': '<22 bytes>'}
DEBUG: ('10.0.2.111', 57396) - ASGI [3] Completed

...

DEBUG: ('10.0.2.111', 57396) - Disconnected
... The background task is completed.
DEBUG: ('10.0.2.111', 57396) - ASGI [2] Completed
DEBUG: ('10.0.2.111', 57397) - Disconnected
DEBUG: ('10.0.2.111', 57405) - Connected

...
The application goes on, with requests and completed background tasks.
At some point, during the execution of a background task:

INFO: Shutting down
DEBUG: ('10.0.2.111', 57568) - Disconnected
DEBUG: ('10.0.2.111', 57567) - Disconnected
INFO: Waiting for background tasks to complete. (CTRL+C to force quit)
DEBUG: ('10.0.2.111', 57567) - ASGI [6] Completed
INFO: Waiting for application shutdown.
DEBUG: None - ASGI [1] Sent {'type': 'lifespan.shutdown'}
DEBUG: None - ASGI [1] Received {'type': 'lifespan.shutdown.complete'}
DEBUG: None - ASGI [1] Completed
INFO: Finished server process [922]

I really don't get why this happens. I have no idea what to try in order to fix it.

My code looks like this.

#!/usr/bin/env python3.7
import time
from fastapi import FastAPI, BackgroundTasks
import uvicorn
from starlette.responses import JSONResponse
import my_imports_from_project

analysis_api = FastAPI()

@analysis_api.get("/")
def root():
    return {"message": "root"}


@analysis_api.get("/test")
def test():
    return {"message": "test"}

@analysis_api.get("/run")
def run(name: str, background_task: BackgroundTasks):
    try:
        some_checks(name)
    except RaisedExceptions:
        body = {"running": False,
                "name": name,
                "cause": "Not found in database"}
        return JSONResponse(status_code=400, content=body)
    body = {"running": True,
            "name": name}
    background_task.add_task(run_analysis, name)
    return JSONResponse(status_code=200, content=body)


if __name__ == "__main__":
    uvicorn.run("api:analysis_api", host="0.0.0.0", log_level="debug")

Have you tried running uvicorn through gunicorn? gunicorn might keep the process up — Will, Jul 15 '19 at 09:52
I'm wondering if the completion of the background tasks triggers a shutdown request somehow — Will, Jul 15 '19 at 09:54
About gunicorn: * It happens the same exact thing but at least gunicord re-starts the server. About the task triggering the shutdown: * I do not know, it just makes computation and saves the results in a database. Moreover, some tasks are successful and just one shuts the server (all tasks are equal) — Davide Melli, Jul 15 '19 at 10:12

score 7 · Accepted Answer · answered Jul 17 '19 at 09:46

This is how I solved the whole problem.

I think that the problem was that my tasks spawn some processes in order to perform computations. So, instead of using FastApi background_task, I am now using multiprocessing.Process(). This solves it.

As pointed out from the guys from FastApi, this solution might not scale well if the project becomes big and complex. In that case it is highly suggested to use something like a message queue + task running (as suggested on FastApi site.

However, for small projects the solution with multiprocessing.Process or subprocess.Popen is totally fine.

Using an external queue will both let you control the number of consumers/subscribers/threads in your app instance processing work in parallel and let you deploy multiple instances of your application that can both add to queue and work on queue. In addition it will enable automatic restarting/resuming queued work items if an instance handling them crashes. — Danny Varod, Dec 20 '21 at 17:08

Uvicorn server shutting down unexpectedly

1 Answers1