Installing pip using ArcGIS-installed Python 2.7

Question

I'm trying to install Scrapy for Python 2.7 on Windows 8.1 and I understand that I first need pip to be installed. Since I have Python installed through ArcGIS 10.2, I think that I need to install pip under C:\Python27\ArcGIS10.2\lib\site-packages. Once pip is installed in that directory I receive the error code:

>>> import pip
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python27\ArcGIS10.2\lib\site-packages\pip\__init__.py", line 10, in <module>
    from pip.util import get_installed_distributions, get_prog
  File "C:\Python27\ArcGIS10.2\lib\site-packages\pip\util.py", line 18, in <module>
    from pip._vendor.distlib import version
  File "C:\Python27\ArcGIS10.2\lib\site-packages\pip\_vendor\distlib\version.py", line 14, in <module>
    from .compat import string_types
  File "C:\Python27\ArcGIS10.2\lib\site-packages\pip\_vendor\distlib\compat.py", line 38, in <module>
    from HTMLParser import HTMLParser
  File "C:\Python27\ArcGIS10.2\lib\HTMLParser.py", line 47, in <module>
    """, re.VERBOSE)
  File "C:\Python27\ArcGIS10.2\lib\re.py", line 190, in compile
    return _compile(pattern, flags)
  File "C:\Python27\ArcGIS10.2\lib\re.py", line 242, in _compile
    raise error, v # invalid expression
error: nothing to repeat

I've also installed pip in C:\Python27\lib\site-packages. However, when it is only installed in that directory PyScripter doesn't recognize that it is installed. Does anyone have any suggestions?

Answer 1

I have a similar setup (Python installed through ArcGIS 10.2, but on machines running Windows 7 not 8.1). I used PIP to install another package (birdy instead of scrapy) and got it working. I think your problem may be trying to work from inside a Python interpreter instead of from the command line (oh, ye mighty Unix users with your ever-powerful command line). Here's what worked for me:

1. Go to http://pip.readthedocs.org/en/latest/installing.html
2. Download the get-pip.py file and place it in your python folder, e.g.: C:\python27\arcgis10.2\
3. Start a command prompt (Start Menu >> Accessories >> Command Prompt)
4. Change directories to python folder by entering: cd c:\python27\arcgis10.2
5. Install PIP by entering: python get-pip.py
6. Change directories into the scripts folder by entering: cd scripts
7. Use pip to install your package (e.g. scrapy) by entering: pip install scrapy

If this works, you should be able to go into Python now and import scrapy. This worked for me on every computer in my lab... just not on my own laptop... will be writing up my own question for that soon (arghh!).

Answer 2

I had the same problem, and I solved it by doing a really CLEAN reinstall of python.

My version of ArcGIS has gone up and down from 10.2.2 to 10.3 to 10.2 to 10.2.2 again. When ArcGIS (or just the Python features) is removed via Add/Remove Programs, most of the files from C:\Python27 will be removed. However, this will not remove the python dll from your system folder. Depending on your Window OS, it will be either:

• C:\Windows\System32\python27.dll
• C:\Windows\SysWOW64\python27.dll

Remove the DLL manually, and then do the python install again.

This should give your a really clean install of python, then run your get-pip.py again, and pip should work as expected!

P.S. The removing of the dll is really important if you are downgrading your python. As the python27.dll seem to get replaced if python is upgraded, but not when it is downgraded. Hence, there is incompatibility between python27.dll with the scripts in C:\Python27.

Answer 3

Modify the file HTMLParser.py as following (C:\Python27\ArcGIS10.2\lib\HTMLParser.py for me):

Before :

locatestarttagend = re.compile(r"""
  <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
  (?:[\s/]*                          # optional whitespace before attribute name
    (?:(?<=['"\s/])[^\s/>][^\s/=>]*  # attribute name
      (?:\s*=+\s*                    # value indicator
        (?:'[^']*'                   # LITA-enclosed value
          |"[^"]*"                   # LIT-enclosed value
          |(?!['"])[^>\s]*           # bare value
         )
       )?(?:\s|/(?!>))*
     )*
   )?
  \s*                                # trailing whitespace
""", re.VERBOSE)

After :

locatestarttagend = re.compile(r"""
  <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
  (?:[\s/]*                          # optional whitespace before attribute name
    (?:(?<=['"\s/])[^\s/>][^\s/=>]*  # attribute name
      (?:\s*=+\s*                    # value indicator
        (?:'[^']*'                   # LITA-enclosed value
          |"[^"]*"                   # LIT-enclosed value
          |(?!['"])[^>\s]*           # bare value
         )
       )?(?:\s|/(?!>))*
     )*
   )  # >>>>>>>>>>>>>>>>>>>>>>>>>>> Remove the ? <<<<<<<<<<<<<<<<<<
  \s*                                # trailing whitespace
""", re.VERBOSE)

Then use pip. I don't know what would be affected by this modification. It may be safer to add the interrogativ point after you used pip