3

I am trying to install psutil with the command pip install -U psutil and that gives me the error:

Cannot uninstall 'psutil'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

It seems like this is a known issue in pip with versions > 10, and I understand that part (I currently have pip 18). But I just found that I can solve it by directly doing a pip install psutil without using the Upgrade flag. I was wondering if there is a reasoning behind that. My initial sense is that in the first case, where pip tries to upgrade, it first tries to remove the package, which it cannot, but in the latter case it tries to install directly, and hence does not get the error. My question is does it still not have to remove the package first and install (when not using the Upgrade flag), or why specifically is it that pip gives an error with an Upgrade flag but no error without it.

EDIT: So, I tried to run pip install -v psutil as hoefling suggested, and I got a whole bunch of text, as opposed to saying that requirements already met, which means that psutil didn't get installed in the first place. I tried to figure this a bit, and this is what I understand so far: I was running inside a python virtualenv and installing it by means of pip -U -r requirements.txt where requirements.txt contains a bunch of packages including psutil. When I remove the -U flag, it skips installing psutil, and jumps over to other packages. Which raises another question, whether this is how pip is supposed to behave when there is no -U flag. Its interesting that the first time, when its installing the packages with the -U flag, it looks inside the main python installation instead of the virtual environment one, and when the -U flag is removed it doesn't do that and skips entirely.

QPTR
  • 1,620
  • 7
  • 26
  • 47
  • First, are you on a Mac, and using Apple's pre-installed Python 2.7, with `psutil` installed as part of Apple's weird `Extras` thing, or did you actually `easy_install psutil` or similar? – abarnert Aug 06 '18 at 21:16
  • @abarnert I am on `Ubuntu 18.04` and used `pip` instead of `easy_install` with `python2.7` – QPTR Aug 06 '18 at 21:23
  • Then I don't know the _exact_ problem, but it's going to be something parallel to the Apple problem, so… see my answer and see if it helps or not. – abarnert Aug 06 '18 at 21:28
  • Are you sure `pip install psutil` actually _does_ install something? `pip` should just tell you `psutil already installed; skipping`, effectively doing nothing. – hoefling Aug 06 '18 at 21:29
  • Did you use a really old version of `pip`, like 1.5.something, to install it originally? `setuptools` has broken backward compatibility a couple of times, but I wouldn't think you'd see that with anything that came with Ubuntu 18.04… – abarnert Aug 06 '18 at 21:32
  • @abarnert Nah, I never actually installed it originally, at least not intentionally, my guess is that it came pre-installed with `Ubuntu 18.04` – QPTR Aug 06 '18 at 21:35
  • @hoefling It definitely got installed. – QPTR Aug 06 '18 at 21:35
  • 2
    You should check whether there are now two copies of `psutil` in two different locations on `sys.path`. As long as the new version is the one in the earlier `sys.path` location, things should just work (and the reason is some variation of what's in my answer, although I don't know the Ubuntu 18.04 specific details). If the old version isn't there at all anymore, something different is going on. – abarnert Aug 06 '18 at 21:36
  • @QPTR can you add the output of `pip install -v psutil` command? – hoefling Aug 06 '18 at 21:42
  • @hoefling See here: https://pastebin.com/MEWa1VQG Also, check the edit I made above in the post. – QPTR Aug 06 '18 at 22:25
  • @abarnert The only installation I can find is inside `/usr/lib/python2.7/dist-packages` and not the virtualenv, which raises an interesting question, why does it look at my main installation when I am using the -U flag, instead of the one inside the virtual environment (see the edit above.) – QPTR Aug 06 '18 at 22:37

1 Answers1

1

There are some setups where you have a bunch of packages installed somewhere that isn't the normal install location for setuptools, and comes after the normal install location on sys.path.

Probably the most common of these setups is Apple's pre-installed Python 2.7, so I'll use it as an example. Even if that isn't your setup, it will be hopefully still be instructive.

Apple includes an Extras directory (at /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python), with a bunch of third-party packages that either Apple's own tools need, or that Apple thought you might want (back when Apple cared about providing the best Python experience of any platform).

For example, on macOS 10.13, that directory will include NumPy 1.8.0.

These packages are all installed as distribute-style eggs.

(Some linux distros do, or at least used to do, similar things, with Python packages built as RPM/DEB/etc. packages, which go into adistutils directory, unlike things you install via pip or manually, which go into a setuptools directory. The details are a bit different, but the effects, and the workaround, end up being the same.)

If you install pip, and then try to pip install -U numpy or pip uninstall numpy, pip will see the distribute-style numpy-1.8.0rc1-py2.7.egg-info file and refuse to touch it for fear of breaking everything.

If you just pip install numpy, however, it will look only in the standard site-packages installation location used by setuptools, /Library/Python/2.7/site-packages, see nothing there, and happily install a modern version of NumPy for you.

And, because /Library/Python/2.7/site-packages comes before /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python on your sys,path, the new NumPy will hide the ancient NumPy, and everything will just work as intended.

There can be a few problems with this. Most notably, if you try to install something which isn't included in Extras itself, but which has a dependency that is included in Extras, it may fail with mysterious and hard-to-debug errors. For example, on macOS 10.12, pip install pandas will throw a bunch of errors at you about not being able to upgrade dateutil, which you didn't even know you were trying to do. The only thing you can do is look at the dependencies for pandas, see which ones are pre-installed in Extras, and manually pip install shadowing versions of all of them.

But, for the most part, it works.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • @abarnet Thanks so much for this answer, its very illuminating. In light of what @hoelfling just suggested, I feel like there is something else happening with the `-U` flag, do you think you could see the above `EDIT` I made. – QPTR Aug 06 '18 at 22:26