175

The new pip dependency resolver that was released with version 20.3 takes an inappropriately long time to install a package. On our CI pipeline yesterday, a docker build that used to take ~10 minutes timed out after 1h of pip installation messages like this (almost for every library that is installed by any dependency there is a similar log output):

INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while.
  Downloading setuptools-50.0.0-py3-none-any.whl (783 kB)
  Downloading setuptools-49.6.0-py3-none-any.whl (803 kB)
  Downloading setuptools-49.5.0-py3-none-any.whl (803 kB)
  Downloading setuptools-49.4.0-py3-none-any.whl (803 kB)
  Downloading setuptools-49.3.2-py3-none-any.whl (790 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading setuptools-49.3.1-py3-none-any.whl (790 kB)
  Downloading setuptools-49.3.0-py3-none-any.whl (790 kB)
  Downloading setuptools-49.2.1-py3-none-any.whl (789 kB)
  Downloading setuptools-49.2.0-py3-none-any.whl (789 kB)
  Downloading setuptools-49.1.3-py3-none-any.whl (789 kB)
  Downloading setuptools-49.1.2-py3-none-any.whl (789 kB)
  Downloading setuptools-49.1.1-py3-none-any.whl (789 kB)
  Downloading setuptools-49.1.0-py3-none-any.whl (789 kB)
  Downloading setuptools-49.0.1-py3-none-any.whl (789 kB)
  Downloading setuptools-49.0.0-py3-none-any.whl (789 kB)
  Downloading setuptools-48.0.0-py3-none-any.whl (786 kB)
  Downloading setuptools-47.3.2-py3-none-any.whl (582 kB)
  Downloading setuptools-47.3.1-py3-none-any.whl (582 kB)
  Downloading setuptools-47.3.0-py3-none-any.whl (583 kB)
  Downloading setuptools-47.2.0-py3-none-any.whl (583 kB)
  Downloading setuptools-47.1.1-py3-none-any.whl (583 kB)
  Downloading setuptools-47.1.0-py3-none-any.whl (583 kB)
  Downloading setuptools-47.0.0-py3-none-any.whl (583 kB)
  Downloading setuptools-46.4.0-py3-none-any.whl (583 kB)
  Downloading setuptools-46.3.1-py3-none-any.whl (582 kB)
  Downloading setuptools-46.3.0-py3-none-any.whl (582 kB)
  Downloading setuptools-46.2.0-py3-none-any.whl (582 kB)
  Downloading setuptools-46.1.3-py3-none-any.whl (582 kB)
  Downloading setuptools-46.1.2-py3-none-any.whl (582 kB)
  Downloading setuptools-46.1.1-py3-none-any.whl (582 kB)
  Downloading setuptools-46.1.0-py3-none-any.whl (582 kB)
  Downloading setuptools-46.0.0-py3-none-any.whl (582 kB)
  Downloading setuptools-45.3.0-py3-none-any.whl (585 kB)
  Downloading setuptools-45.2.0-py3-none-any.whl (584 kB)
  Downloading setuptools-45.1.0-py3-none-any.whl (583 kB)
  Downloading setuptools-45.0.0-py2.py3-none-any.whl (583 kB)
  Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
  Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB)
  Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
  Downloading setuptools-43.0.0-py2.py3-none-any.whl (583 kB)
  Downloading setuptools-42.0.2-py2.py3-none-any.whl (583 kB)
  Downloading setuptools-42.0.1-py2.py3-none-any.whl (582 kB)
  Downloading setuptools-42.0.0-py2.py3-none-any.whl (582 kB)
  Downloading setuptools-41.6.0-py2.py3-none-any.whl (582 kB)
  Downloading setuptools-41.5.1-py2.py3-none-any.whl (581 kB)
  Downloading setuptools-41.5.0-py2.py3-none-any.whl (581 kB)
  Downloading setuptools-41.4.0-py2.py3-none-any.whl (580 kB)
  Downloading setuptools-41.3.0-py2.py3-none-any.whl (580 kB)
  Downloading setuptools-41.2.0-py2.py3-none-any.whl (576 kB)
  Downloading setuptools-41.1.0-py2.py3-none-any.whl (576 kB)
  Downloading setuptools-41.0.1-py2.py3-none-any.whl (575 kB)
  Downloading setuptools-41.0.0-py2.py3-none-any.whl (575 kB)
  Downloading setuptools-40.9.0-py2.py3-none-any.whl (575 kB)
  Downloading setuptools-40.8.0-py2.py3-none-any.whl (575 kB)
  Downloading setuptools-40.7.3-py2.py3-none-any.whl (574 kB)
  Downloading setuptools-40.7.2-py2.py3-none-any.whl (574 kB)
  Downloading setuptools-40.7.1-py2.py3-none-any.whl (574 kB)
  Downloading setuptools-40.7.0-py2.py3-none-any.whl (573 kB)
  Downloading setuptools-40.6.3-py2.py3-none-any.whl (573 kB)
  Downloading setuptools-40.6.2-py2.py3-none-any.whl (573 kB)
  Downloading setuptools-40.6.1-py2.py3-none-any.whl (573 kB)
  Downloading setuptools-40.6.0-py2.py3-none-any.whl (573 kB)
  Downloading setuptools-40.5.0-py2.py3-none-any.whl (569 kB)
  Downloading setuptools-40.4.3-py2.py3-none-any.whl (569 kB)
  Downloading setuptools-40.4.2-py2.py3-none-any.whl (569 kB)
  Downloading setuptools-40.4.1-py2.py3-none-any.whl (569 kB)
  Downloading setuptools-40.4.0-py2.py3-none-any.whl (568 kB)
  Downloading setuptools-40.3.0-py2.py3-none-any.whl (568 kB)

I am quite confused whether we are using the new pip resolver correctly, especially since

- Substantial improvements in new resolver for performance, output and error messages, avoiding infinite loops, and support for constraints files.

The behavior seen is described as backtracking in the release notes. I understand why it is there. It specifies that I can use a constraint file (looks like a requirements.txt) that fixes the version of the dependencies to reduce the runtime using pip install -c constraints.txt setup.py.

What is the best way to produce this constraints file? Currently, the best way I can think of is running pip install setup.py locally in a new virtual environment, then using pip freeze > constraints.txt. However, this still takes a lot of time for the local install (it's been stuck for about 10 minutes now). The notes do mention that This means the “work” is done once during development process, and so will save users this work during deployment.

With the old dependency resolver, I was able to install this package in less than a minute locally.

What is the recommended process here?

Edit: I just found out that some of the dependencies are pointing directly to out internal gitlab server. If I instead install directly from our internal package registry, it works in a couple of minutes again.

Simulant
  • 19,190
  • 8
  • 63
  • 98
RunOrVeith
  • 4,487
  • 4
  • 32
  • 50
  • 1
    I have the same issue. Your edit seems interesting, can you add some guidance on how you detected the dependency link to your gitlab and how you fixed it ? – ldenisey Dec 08 '20 at 10:01
  • 1
    Inside the setup.py the dependency was something like `my_lib @ git+ssh://git@gitlab.our-company.com/libraries/my_lib.git@v1.2.0`, so if you did not manually put something there to point to your own gitlab server you will not have this issue – RunOrVeith Dec 08 '20 at 12:26
  • 1
    Thank you. Indeed, that is not our case. – ldenisey Dec 08 '20 at 14:16
  • 2
    I will probably downgrade to pip 19 again.. – coderboi Feb 25 '21 at 22:34
  • 2
    unfortunately pip 19 produces bandit security warnings :/ so, no longer an option :( A shame, since pip 19 worked really nicely. I really dislike hte new change. There should at least be an option like `--no-backtracking`. When I ask to install package A, without specifying a version, and that package is not already installed, I expect pip to install the latest version of A, or die trying. I dont expect it to install some version of that package from 2 years ago. – Hugh Perkins Sep 21 '21 at 22:40

9 Answers9

123

Latest update (2022-02)

There seems to be major update in pip just few days old (version 22.0, release notes + relevant issue on github).

I haven't tested it in more detail but it really seems to me that they optimized installation order calculation in complex case in such way that it resolves many issues we all encountered earlier. But I will need more time to check it.

Anyway, the rest of this answer is still valid and smart requirements pinning suitable for particular project is a good practice imo.


Since I encountered similar issue I agree this is quite annoying. Backtracking might be useful feature but you don't want to wait hours to complete with uncertain success.

I found several option that might help:

  • Use the old resolver (--use-deprecated=legacy-resolver) proposed in the answer by @Daniel Davee, but this is more like temporary solution than a proper one.
  • Skip resolving dependencies with --no-deps option. I would not recommend this generally but in some cases you can have a working set of packages versions although there are some conflicts.
  • Reduce the number of versions pip will try to backtrack and be more strict on package dependencies. This means instead of putting e.g. numpy in my requirements.txt, I could try numpy >= 1.18.0 or be even more strict with numpy == 1.18.0. The strictness might help a lot.

Check the following sources:

I still do not have a proper answer that would always help but the best practice for requirements.txt seems to "pin" package versions. I found pip-tools that could help you manage this even with constrains.txt (but I am in an experimental phase so I can not tell you more).

Update (2021-04)

It seems author of the question was able to fix the issue (something with custom gitlab server) but I would like to extend this answer since it might be useful for others.

After reading and trying I ended up with pinning all my package versions to a specific one. This really should be the correct way. Although everything can still work without it, there might be cases where if you don't pin your dependencies, your package manager will silently install a new version (when it's released) with possible bugs or incompatibility (this happens to me with dask last this year).

There are several tools which might help you, I would recommend one of these approaches:

Easiest one with pipreqs

  • pipreqs is a library which generates pip requirements.txt file based on imports of any project
  • you can start by pip install pipreqs and runnning just pipreqs in your project root (or eventually with --force flag if your requirements already exists)
  • it will easily create requirements.txt with pinned versions based on imports in your project and versions taken from your environment
  • then you can at any time create new environment based on this requirements.txt

This is really simple tool (you even do not need to write your requirements.txt). It does not allow you to create something complex (might not be a good choice for bigger projects), last week I found one strange behavior (see this) but generally I'm happy with this tool as it usually works perfectly.

Using pip-tools

There are several other tools commonly used like pip-tools, Pipenv or Poetry. You can read more in Faster Docker builds with pipenv, poetry, or pip-tools or Python Application Dependency Management in 2018 (older but seems still valid to me). And it still seems to me that the best option (although it depends on your project/use case) is pip-tools.

You can (this is one option, see more in docs):

  • create requirements.in (the same format as requirements.txt, it's up to you whether you pin some package dependency or not)
  • then you can use it by pip install pip-tools and running pip-compile requirements.in
  • this will generate new requirements.txt file where all versions are pinned, it's clear, what is the origin
  • (Optionally) you can run it with --generate-hashes option
  • then you can (as with pipreqs) at any time create new environment based on this requirements.txt
  • pip-tools offer you --upgrade option to upgrade the final reqs
  • supports layered requirements (e.g. having dev and prod versions)
  • there is integration with pre-commit
  • offers pip-sync tool to update your environment based on requirements.txt

There are few more stuff you can do with it and I really love the integration with pre-commit. This allows you to use the same requirements as before (just with .in suffix) and add pre-commit hook that automatically updates requirements.txt (so you will never experience having different local environment from the generated requirements.txt which might easily happen when you run something manually).

Nerxis
  • 3,452
  • 2
  • 23
  • 39
  • 28
    I already have everything pinned to exact versions and I still get the eternal "pip is looking at multiple versions" message. :( – Peter Bengtsson May 18 '21 at 13:49
  • @PeterBengtsson: Could you share your requirements.txt? – Nerxis May 18 '21 at 15:08
  • https://gist.github.com/peterbe/48ed9016a3f715ea15fa80be983b3263 and I use pip 21.1.1 and type: `pip install -r dev-requirements.txt -r requirements-constraints.txt -r requirements.txt` – Peter Bengtsson May 21 '21 at 20:00
  • 2
    Still just hangs. Tried many times. Every single dependency is pinned exactly. – Peter Bengtsson May 21 '21 at 20:14
  • I just commented your gist here: https://gist.github.com/peterbe/48ed9016a3f715ea15fa80be983b3263#gistcomment-3765908 (we can continue discussion there) – Nerxis Jun 02 '21 at 07:29
  • `--use-feature=fast-deps` - Can speeds things up a lot as well. – GrimSqueaker Sep 13 '21 at 09:30
  • @Nerxis what does pipreqs do that pip freeze does not? – sureshvv Sep 27 '21 at 07:45
  • 1
    @sureshvv check this https://github.com/bndr/pipreqs#why-not-pip-freeze – Nerxis Sep 27 '21 at 08:07
  • Perhaps, "why not freeze" was more true when it was first written than now. Although freeze does include things that are not directly dependent on packages that are used, if the output of freeze is used as a constraints file rather than a requirements file, then it will not cause anything unnecessary to be installed. – Steve Jorgensen Nov 01 '21 at 19:40
  • @SteveJorgensen Good point, thanks. But I still see the issue with pip freeze as it does not offer you a possibility to export only those dependecies that are relevant for your project. So, yes, I can imagine you use your own requirements file and than you can use `pip freeze` output for constraints file. However, I think using tools that already combine those things together and make it much easier is better way to go. – Nerxis Nov 02 '21 at 08:57
  • Rant: I wish someone would do something to fix the new resolver so we don't have to spend hours hunting for workarounds, dealing with this issue etc. Like, what I want is an option, using the new resolve of `--do-not-backtrack`. If I ask to install a package, I expect the *latest* version to *always* be installed, unless I explicitly ask otherwise. This would solve a lot of pain... – Hugh Perkins Dec 07 '21 at 12:49
  • @HughPerkins: See the latest update of my question. It seems that new version of pip is doing much better work. It does not bring what you wish but it might help you in some cases. – Nerxis Feb 16 '22 at 12:16
  • would be really nice if they solved these issues before bricking critical python infrastructure – Kevin R. May 15 '22 at 23:10
71

So they are changing the resolver, this seems to be a bug. What worked for was using the old resolver, by using the flag

--use-deprecated=legacy-resolver

This will work until pip 21.0 apparently.

source https://github.com/pypa/pip/issues/9215

Sunil Garg
  • 14,608
  • 25
  • 132
  • 189
Daniel Davee
  • 756
  • 4
  • 3
  • 13
    I am specifically asking for solutions not using the old resolver – RunOrVeith Dec 04 '20 at 16:59
  • 5
    @Bromide, message and error trace has already been provided by the OP. What else do you need? And it's not just him, multiple people including myself are getting the same error and we are looking for solutions to fix this issue. – hky404 Dec 08 '20 at 07:19
  • This is actually really useful because when you switch to the old resolver, it will print an error for unresolved dependencies and tell exactly which version pins are causing the problem. – Edward D'Souza May 10 '21 at 20:36
  • 4
    FYI, `--use-deprecated=legacy-resolver` is still supported in pip 21.1.2 – Steven Ensslen May 26 '21 at 21:21
  • 1
    Thank you for adding this, i spent hours waiting for pip install to finish. Using the old resolver it finished in 2 minutes. – Moulde Jun 18 '21 at 14:42
  • Unfortunately, pip < 21.0 now fails safety checks, so using the old resolver is no longer an option using a version of pip that _does_ pass safety checks. I'm fighting with this right now. I'm using frozen constraints, so there should be no resolution to do. Even though pip is only looking at 1 version of each package (after saying it is looking at multiple) the `pip install` gets progressively slower as it runs until it becomes intolerably slow. – Steve Jorgensen Nov 01 '21 at 19:44
9

Had similar issue and wanted to report the solution which worked for me. I had to update the pip version using

pip install --upgrade pip

Actually I was building a docker image so I've added the line

RUN pip install --upgrade pip

Right before installing the requirements. In the time of writing this answer the installed pip version is 22.0.3. Doing so fixed the problem.

antonpuz
  • 3,256
  • 4
  • 25
  • 48
2

In my case it never finished. The reason was updating djangorestframework to a version incompatible with django. I wanted it to install djangorestframework-3.12.4, which depends on django >= 2.2, and also had django==2.0.2 in requirements.txt.

x-yuri
  • 16,722
  • 15
  • 114
  • 161
2

I have the same problem and the provided solutions didn't work for me, even though I have pinned all the dependencies.

So I looked at the first package that caused PIP to backtrack in the console output (the package before the first warning: "INFO: pip is looking at multiple versions of ___ to determine which version is compatible with other requirements. This could take a while.") and remove it from requirements.txt. This fixed the problem for me.

Gnut
  • 541
  • 1
  • 5
  • 19
1

I have pinned all of my dependencies. You will have to pin them later or now so better to do it now. I am on pip 20.3.1

If we don't pin any of our dependency, the new dependency resolver algorithm checks every single available version against each dependency from the latest to the initial version tags.

Umar Asghar
  • 3,808
  • 1
  • 36
  • 32
1

Downgrade your pip to version 20.x this is the issue you'll face in pip version 21.x

  • 3
    Please improve your reply by adding some relevant information, exact steps etc. – aroyc Dec 05 '21 at 07:35
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 05 '21 at 09:17
  • Not an option for myself, since 20.x has security issues that are flagged by `bandit`. – Hugh Perkins Dec 05 '21 at 14:32
  • 1
    I've spent the last few weeks working on fixing extremely slow builds all of my automated test runners. Downgrading pip itself is the only solution that's been consistently successful for me. Pip should add a `--no-backtracking` command line option to completely disable this feature (that's default). – Seth Feb 02 '23 at 19:21
1

In my case we used a virtual environment and it helped to use python 3.10.x instead of 3.8.3. E.g.

python3.10 -m venv venv
source venv/bin/activate
pip install .

Pip was already upgraded to 22.3.1.

kaikun
  • 98
  • 10
-3

Even though all my packages in requirements.txt were version fixed, the issue was still present. I just updatem them to their latest versions instead. Of course this brings an extra effort on guaranteeing nothing breaks, but it seemed to solve the issue for me.

Braiam
  • 1
  • 11
  • 47
  • 78
  • This is not a sustainable solution since in you case it was just luck, that it worked after updating the versions. – aoez Feb 27 '23 at 12:01