Why does installing a python package break setuptools and causes pkg_resources to not be found?

Question

This is partly a question, partly my own findings on what I found to be the issue when I encountered this error:

(cdbak)USER-MBP-2:.virtualenvs <YOUR_USER_NAME>$ pip 
Traceback (most recent call last):
    File "/Users/<YOUR_USER_NAME>/.virtualenvs/cdbak/bin/pip", line 6, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

This issue arose when I tried to install pypsum via pip in my virtual environment for use with django.

(cdbak)USER-MBP-2:.virtualenvs <YOUR_USER_NAME>$ pip install pypsum

I have been working in virtual environments, so I was fortunate that after it broke, I could just reset my virtual environment with the script I had written.

I copied the output from the installation process and started looking closer into it and it seems like there is something that goes wrong in setuptools.

The installation process tries to build the package but it does not find build_py in the setuptools which causes it to 'patch' the setuptools installation by renaming the currently installed setuptools.

This is the part where I think funky stuff starts to happen:

Setuptools installation detected at /Users/<YOUR_USER_NAME>/.virtualenvs/cdbak/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
Egg installation
Patching...
Renaming /Users/<YOUR_USER_NAME>/.virtualenvs/cdbak/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg into /Users/<YOUR_USER_NAME>/.virtualenvs/cdbak/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg.OLD.1321360113.04

And then it goes on to try and install another version of setuptools or so it seems:

After install bootstrap.
Creating /Users/<YOUR_USER_NAME>/.virtualenvs/cdbak/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info
Creating /Users/<YOUR_USER_NAME>/.virtualenvs/cdbak/lib/python2.7/site-packages/setuptools.pth

But it does not seem to install the setuptools package correctly in its current location and then results in a missing pkg_resources module (In fact, it is missing a lot of other things too)

[Setup]

OS: Mac OS X Lion
Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05)
virtualenv v1.6.1

Fresh virtual environment using virtualenvwrapper using requirements file to install these packages:

mercurial==1.9.3
Django>=1.3.1
MySQL-python>=1.2.3
Sphinx
wsgiref
pylint
yolk
dbgp
django-debug-toolbar
south

I was able to use pip to install other packages just fine but for some reason it seems to break with this installation.

[Question]
- Do people think this is a mistake with the installation process of this package?
- Or, is it a mistake with setuptools?
- Or, am I just installing it incorrectly?

[Extra Note]
I can attach the file with the entire output but its a long file and I decided to only extract the segments I felt were relevant. If you would like to view the full file, I can upload that too.

Which version of OS X? Which Python instance did you use to create your `virtualenv` (e.g. what is the version info when launching `python`)? — Ned Deily, Nov 19 '11 at 00:56
Not the OP but I was able to reproduce on OS X Lion and python 2.7. — wilbbe01, Nov 19 '11 at 01:34

score 3 · Accepted Answer · answered Nov 19 '11 at 02:41

I'm not sure about all of your problems but there is at least one problem caused by the loremipsum package that is a dependency for the pypsum package. For some reason, in the setup.py file of loremipsum, the author includes specific requirements for the distribute package:

egg = {
    'name': name,
    'version': module.__version__,
    'author': author,
    'author_email': email.strip('<>'),
    'url': url,
    'description': "A Lorem Ipsum text generator",
    'long_description': long_description,
    'classifiers': module.__classifiers__,
    'keywords': ['lorem', 'ipsum', 'text', 'generator'],
    'setup_requires': ['distribute'],
    'install_requires': ['distribute'],
    'packages': [name],
    # 'package_dir': {'': '.'},
    # 'package_data': {'': 'default/*.txt'},
    # 'data_files': [(name, ('default/dictionary.txt', 'default/sample.txt'))],
    'include_package_data': True,
    'test_suite': 'tests.suite' }

distribute, as you may know, is a fork of the setuptools package; there's a long history behind this. Since distribute is supposed to be an almost plug-compatible replacement for setuptools, it will try to masquerade as setuptools and disable any existing setuptools already installed in that Python instance. Thus, putting distribute in a setup.py file as a requirement is usually not a good idea. By default, virtualenv will install a version of setuptools but it does have an option to use distribute instead. The Apple-supplied system Pythons in OS X 10.6 and 10.7 already come with versions of setuptools pre-installed and, because they are in non-standard system directories, cannot be so easily patched around. A simple workaround when using virtualenv on OS X seems to be to also use its no-site-packages option which will prevent the setuptools version from the system Python interfering with the required distribute in the virtualenv. No doubt the confusion between distribute and setuptools is causing the problems seen with pkg_resources since it is also supplied by both of them.

So try re-creating your virtualenv this way:

virtualenv --distribute --no-site-packages /path/to/ve

That will also have the side-effect of not including the third-party packages that Apple ships with the system Python. You can add them back in with PYTHONPATH if you really need them but it's probably better to install separate versions.

Hi Ned, thank you for the response. I did install virtualenv using a boostrap virtualenv and with exactly the flags mentioned above in your post. I then proceeded to install virtualenvwrapper and used that to create the virtual environment with no-site-packages too but without distribute. I wonder if I should have used --distribute too? As for my concern, would you recommend I forward this issue to loremipsum as a potential issue or is this more of a python installation procedure issue or just me not installing virtual environments correctly and requires no bug reporting? — MD3Sum, Nov 19 '11 at 07:46
I believe you will need to create the virtualenv with distribute because that will (1) install the distribute clone of setuptools that pip and virtualenv require and (2) satisfy the package requirement for `loremipsum`. Another way to avoid this is to not use the Apple-supplied system Python and thus avoid the built-in setuptools. If it all works as I predict, I would forward the issue to loremipsum. But the bigger lesson here is to avoid having multiple copies of setuptools/distribute in use in one Python instance or virtualenv. — Ned Deily, Nov 19 '11 at 07:56
Hi Ned, I tried your suggestion on a clean _virtualenv_ with the **--distribute --no-site-packages** flags and it worked. I had added --no-site-packages to my .bash_profile file in the arg: **VIRTUALENVWRAPPER_VIRTUALENV_ARGS** for _virtualenvwrapper_ but I forgot to add the --distribute flag by default. Having done that now, it works. I will send the creators of lorempipsum this link in any case to give them some feedback. Thanks again. — MD3Sum, Nov 19 '11 at 08:16

Why does installing a python package break setuptools and causes pkg_resources to not be found?

1 Answers1

Linked