1

I'm trying to install shub, the Scrapinghub command line tool in OSX 10.11.6 (El Capitan) via pip. The installation script downloads the required modules and at some point returns the following error:

OSError: [Errno 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/man'

I used the following command: 

$ sudo -H pip install shub --ignore-installed six

and received the following traceback: 

Collecting shub
  Using cached shub-2.7.0-py2.py3-none-any.whl
Collecting six
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting retrying (from shub)
  Using cached retrying-1.3.3.tar.gz
Collecting tqdm (from shub)
  Using cached tqdm-4.19.2-py2.py3-none-any.whl
Collecting pip (from shub)
  Using cached pip-9.0.1-py2.py3-none-any.whl
Collecting PyYAML (from shub)
  Using cached PyYAML-3.12.tar.gz
Collecting docker-py (from shub)
  Using cached docker_py-1.10.6-py2.py3-none-any.whl
Collecting click (from shub)
  Using cached click-6.7-py2.py3-none-any.whl
Collecting requests (from shub)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting scrapinghub>=1.9.0 (from shub)
  Using cached scrapinghub-2.0.1-py2.py3-none-any.whl
Collecting backports.ssl-match-hostname>=3.5; python_version < "3.5" (from docker-py->shub)
  Using cached backports.ssl_match_hostname-3.5.0.1.tar.gz
Collecting ipaddress>=1.0.16; python_version < "3.3" (from docker-py->shub)
  Using cached ipaddress-1.0.18-py2-none-any.whl
Collecting docker-pycreds>=0.2.1 (from docker-py->shub)
  Using cached docker_pycreds-0.2.1-py2.py3-none-any.whl
Collecting websocket-client>=0.32.0 (from docker-py->shub)
  Using cached websocket_client-0.44.0-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests->shub)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests->shub)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->shub)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->shub)
  Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Installing collected packages: six, retrying, tqdm, pip, PyYAML, urllib3, idna, chardet, certifi, requests, backports.ssl-match-hostname, ipaddress, docker-pycreds, websocket-client, docker-py, click, scrapinghub, shub
  Running setup.py install for retrying ... done
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 377, in move_wheel_files
    clobber(source, dest, False, fixer=fixer, filter=filter)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 316, in clobber
    ensure_dir(destdir)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/man'

Any thoughts as to what is causing this problem? Thanks in advance.

EDIT:

The problem was solved by installing Homebrew. The instructions are on this link: https://docs.scrapy.org/en/latest/intro/install.html#mac-os-x

After Homebrew was installed, I installed shub with the following command: pip install shub and received the following: 

Collecting shub
  Using cached shub-2.7.0-py2.py3-none-any.whl
Requirement already satisfied: retrying in /Library/Python/2.7/site-packages (from shub)
Collecting click (from shub)
  Using cached click-6.7-py2.py3-none-any.whl
Requirement already satisfied: requests in /Library/Python/2.7/site-packages (from shub)
Collecting scrapinghub>=1.9.0 (from shub)
  Using cached scrapinghub-2.0.1-py2.py3-none-any.whl
Requirement already satisfied: pip in /usr/local/lib/python2.7/site-packages (from shub)
Collecting docker-py (from shub)
  Using cached docker_py-1.10.6-py2.py3-none-any.whl
Requirement already satisfied: PyYAML in /Library/Python/2.7/site-packages (from shub)
Requirement already satisfied: tqdm in /Library/Python/2.7/site-packages (from shub)
Requirement already satisfied: six>=1.7.0 in /Library/Python/2.7/site-packages (from shub)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /Library/Python/2.7/site-packages (from requests->shub)
Requirement already satisfied: idna<2.7,>=2.5 in /Library/Python/2.7/site-packages (from requests->shub)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Library/Python/2.7/site-packages (from requests->shub)
Requirement already satisfied: certifi>=2017.4.17 in /Library/Python/2.7/site-packages (from requests->shub)
Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /Library/Python/2.7/site-packages (from docker-py->shub)
Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /Library/Python/2.7/site-packages (from docker-py->shub)
Collecting docker-pycreds>=0.2.1 (from docker-py->shub)
  Using cached docker_pycreds-0.2.1-py2.py3-none-any.whl
Collecting websocket-client>=0.32.0 (from docker-py->shub)
  Using cached websocket_client-0.44.0-py2.py3-none-any.whl
Installing collected packages: click, scrapinghub, docker-pycreds, websocket-client, docker-py, shub
Successfully installed click-6.7 docker-py-1.10.6 docker-pycreds-0.2.1 scrapinghub-2.0.1 shub-2.7.0 websocket-client-0.44.0

Now shub is installed.

NEW PROBLEM: shub has been installed, but none of the shub commands are returning data. When using the shub login command, I receive the following traceback:

Traceback (most recent call last):
  File "/usr/local/bin/shub", line 7, in <module>
    from shub.tool import cli
  File "/usr/local/lib/python2.7/site-packages/shub/tool.py", line 57, in <module>
    command_module = importlib.import_module(module_path)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/site-packages/shub/deploy.py", line 10, in <module>
    import setuptools
  File "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 10, in <module>
    from setuptools.extern.six.moves import filter, map
  File "/usr/local/lib/python2.7/site-packages/setuptools/setuptools/__init__.py", line 160, in <module>
    monkey.patch_all()
  File "/usr/local/lib/python2.7/site-packages/setuptools/setuptools/monkey.py", line 67, in patch_all
    distutils.core.Command = setuptools.Command
AttributeError: 'module' object has no attribute 'Command'

I checked to see if there were any required version of packaging present that i needed, but everything is current.

Any help is greatly appreciated. Thanks in advance.

Rell
  • 11
  • 1
  • 3
  • Are you sure you can put the `--ignore-installed` flag between packages? – Brenden Petersen Oct 13 '17 at 21:16
  • @BrendenPetersen i used the `--ignore-installed` flag when installing scrapy, and it worked. I had to use it because there was already an installed version of six on my machine during the time I was trying to install scrapy. After using that flag, scrapy installed smoothly. – Rell Oct 13 '17 at 21:39
  • Right, but your command is `sudo -H pip install --ignore-installed `. You're installing two packages with that command. And you insert a flag between them. I'm not sure whether that's allowed. Maybe try putting `--ignore-installed` before `shub`? Or if you only want to ignore installed for one package, then issue two separate commands. – Brenden Petersen Oct 13 '17 at 22:08
  • 1
    FWIW, as far as I know HomeBrew Python shouldn't be trying to use ``/Library/Python/2.7/site-packages``. So there is something screwed up with your Python installations. – Graham Dumpleton Oct 15 '17 at 03:15
  • The reason the first attempt failed is that Python packages don't usually install manual pages. One of the packages, not necessarily ``shub`` is trying to install manual pages using a location it is calculating itself. That though is resolving to a protected area (protected by System Integrity Protection/SIP) of the Apple OS file system directories. You could have avoid the issue by using a Python virtual environment. Never a good idea to install packages into main Python except for ``pip`` and ``virtualenv``. – Graham Dumpleton Oct 15 '17 at 03:19
  • @GrahamDumpleton thank you for the tip. I uninstalled all of the initial downloads of python and re-downloaded everything according to the homebrew install directions. Then created the downloaded the `virtualenv` package and pip package via the homebrew installation directions. Now everything works perfectly, including `shub`. It is important to note that `pip3` was used to install everything while in the virtual environment. Thanks for all of the input. Problem solved. – Rell Oct 29 '17 at 03:39

0 Answers0