You shouldn't use any short-lived background task solution either offered by Celery or FastAPI for constant background tasks. (FastAPI, Celery) You should use a separate always-running process for reading Serial ports. You can run it as a single Python daemon process. (You can get the help of process management tools like a supervisord or you can even Dockerize your process if you want.) The most important thing to consider here is what happens when your process faces an unexpected case. You should know what happened exactly. That's where proper exception handling and logging shine, also the process should try to recover without manual intervention if it exited unexpectedly. (That's why you should use process management systems, they have options like auto restart).
If your other background tasks triggered by certain actions will be lightweight and your use case is simple then you can choose to use FastAPI background tasks but it doesn't provide the full functionality that Celery offers, you need to think about that. You may refer to the documentation of both and think if you need the functionalities that Celery offers. (But don't forget that Celery will add an additional complexity and a component to maintain in return what it offers.)
Both Flask and FastAPI are used in production so you can choose any of them and async processing can be achieved in both with the help of the tools like Celery. You may just refer to their docs, think about your use case, and decide.