How do I set a custom gunicorn worker timeout when serving an MLflow model with the "mlflow models serve" CLI?

Question

When serving an MLflow Python model with the "pyfunc" backend (https://github.com/mlflow/mlflow/blob/master/mlflow/pyfunc/backend.py), how can I set a custom gunicorn worker timeout? The default timeout of 60 seconds may be insufficient when serving large models that take a long time to load.

Answer 1

As of MLflow 1.2, you can set a custom gunicorn timeout by specifying the GUNICORN_CMD_ARGS environment variable. The following example serves a model with a worker timeout of 120 seconds

GUNICORN_CMD_ARGS="--timeout 120" mlflow models serve --model-uri /path/to/model

Answer 2

mlflow allows setting these options from the cli:

Example: mlflow models serve ... --timeout 180

Official documentation (mlflow models serve --help):

-t, --timeout TEXT Timeout in seconds to serve a request (default: 60).