The new pip dependency resolver that was released with version 20.3 takes an inappropriately long time to install a package. On our CI pipeline yesterday, a docker build that used to take ~10 minutes timed out after 1h of pip installation messages like this (almost for every library that is installed by any dependency there is a similar log output):
INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while.
Downloading setuptools-50.0.0-py3-none-any.whl (783 kB)
Downloading setuptools-49.6.0-py3-none-any.whl (803 kB)
Downloading setuptools-49.5.0-py3-none-any.whl (803 kB)
Downloading setuptools-49.4.0-py3-none-any.whl (803 kB)
Downloading setuptools-49.3.2-py3-none-any.whl (790 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Downloading setuptools-49.3.1-py3-none-any.whl (790 kB)
Downloading setuptools-49.3.0-py3-none-any.whl (790 kB)
Downloading setuptools-49.2.1-py3-none-any.whl (789 kB)
Downloading setuptools-49.2.0-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.3-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.2-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.1-py3-none-any.whl (789 kB)
Downloading setuptools-49.1.0-py3-none-any.whl (789 kB)
Downloading setuptools-49.0.1-py3-none-any.whl (789 kB)
Downloading setuptools-49.0.0-py3-none-any.whl (789 kB)
Downloading setuptools-48.0.0-py3-none-any.whl (786 kB)
Downloading setuptools-47.3.2-py3-none-any.whl (582 kB)
Downloading setuptools-47.3.1-py3-none-any.whl (582 kB)
Downloading setuptools-47.3.0-py3-none-any.whl (583 kB)
Downloading setuptools-47.2.0-py3-none-any.whl (583 kB)
Downloading setuptools-47.1.1-py3-none-any.whl (583 kB)
Downloading setuptools-47.1.0-py3-none-any.whl (583 kB)
Downloading setuptools-47.0.0-py3-none-any.whl (583 kB)
Downloading setuptools-46.4.0-py3-none-any.whl (583 kB)
Downloading setuptools-46.3.1-py3-none-any.whl (582 kB)
Downloading setuptools-46.3.0-py3-none-any.whl (582 kB)
Downloading setuptools-46.2.0-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.3-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.2-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.1-py3-none-any.whl (582 kB)
Downloading setuptools-46.1.0-py3-none-any.whl (582 kB)
Downloading setuptools-46.0.0-py3-none-any.whl (582 kB)
Downloading setuptools-45.3.0-py3-none-any.whl (585 kB)
Downloading setuptools-45.2.0-py3-none-any.whl (584 kB)
Downloading setuptools-45.1.0-py3-none-any.whl (583 kB)
Downloading setuptools-45.0.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-43.0.0-py2.py3-none-any.whl (583 kB)
Downloading setuptools-42.0.2-py2.py3-none-any.whl (583 kB)
Downloading setuptools-42.0.1-py2.py3-none-any.whl (582 kB)
Downloading setuptools-42.0.0-py2.py3-none-any.whl (582 kB)
Downloading setuptools-41.6.0-py2.py3-none-any.whl (582 kB)
Downloading setuptools-41.5.1-py2.py3-none-any.whl (581 kB)
Downloading setuptools-41.5.0-py2.py3-none-any.whl (581 kB)
Downloading setuptools-41.4.0-py2.py3-none-any.whl (580 kB)
Downloading setuptools-41.3.0-py2.py3-none-any.whl (580 kB)
Downloading setuptools-41.2.0-py2.py3-none-any.whl (576 kB)
Downloading setuptools-41.1.0-py2.py3-none-any.whl (576 kB)
Downloading setuptools-41.0.1-py2.py3-none-any.whl (575 kB)
Downloading setuptools-41.0.0-py2.py3-none-any.whl (575 kB)
Downloading setuptools-40.9.0-py2.py3-none-any.whl (575 kB)
Downloading setuptools-40.8.0-py2.py3-none-any.whl (575 kB)
Downloading setuptools-40.7.3-py2.py3-none-any.whl (574 kB)
Downloading setuptools-40.7.2-py2.py3-none-any.whl (574 kB)
Downloading setuptools-40.7.1-py2.py3-none-any.whl (574 kB)
Downloading setuptools-40.7.0-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.3-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.2-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.1-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.6.0-py2.py3-none-any.whl (573 kB)
Downloading setuptools-40.5.0-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.3-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.2-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.1-py2.py3-none-any.whl (569 kB)
Downloading setuptools-40.4.0-py2.py3-none-any.whl (568 kB)
Downloading setuptools-40.3.0-py2.py3-none-any.whl (568 kB)
I am quite confused whether we are using the new pip resolver correctly, especially since
- Substantial improvements in new resolver for performance, output and error messages, avoiding infinite loops, and support for constraints files.
The behavior seen is described as backtracking in the release notes. I understand why it is there.
It specifies that I can use a constraint file (looks like a requirements.txt) that fixes the version of the dependencies to reduce the runtime using
pip install -c constraints.txt setup.py
.
What is the best way to produce this constraints file? Currently, the best way I can think of is running pip install setup.py
locally in a new virtual environment, then using pip freeze > constraints.txt
. However, this still takes a lot of time for the local install (it's been stuck for about 10 minutes now).
The notes do mention that This means the “work” is done once during development process, and so will save users this work during deployment.
With the old dependency resolver, I was able to install this package in less than a minute locally.
What is the recommended process here?
Edit: I just found out that some of the dependencies are pointing directly to out internal gitlab server. If I instead install directly from our internal package registry, it works in a couple of minutes again.