1

I am trying to install Scrapy on my computer and it has been a while since I used pip to install any libraries. Whenever I type:

pip install scrapy

into my command prompt on Windows 10 I get the error message:

Import error: No module named site

Doing a search I have found the below link:

ImportError: No module named site on Windows

with a similar issue and gone to Advanced Systems Settings and Enviromental variables, adding PYTHONHOME with C:\Python38 and PYTHONPATH with C:\Python38\Lib and C:\Python38\Scripts to both user and system variables. But I am still getting the same error message.

I have also uninstall Python and Anaconda and reinstalled them.

Does anyone know anything else I can try or any other reason pip may not be working?

Christopher Ell
  • 1,878
  • 3
  • 16
  • 25

2 Answers2

1

You may use this way to install scrapy on your windows machine.

From http://doc.scrapy.org/en/latest/intro/install.html

Platform specific installation notes

Windows

Though it’s possible to install Scrapy on Windows using pip, we recommend you to install Anaconda or Miniconda and use the package from the conda-forge channel, which will avoid most installation issues.

Once you’ve installed Anaconda or Miniconda, install Scrapy with:

conda install -c conda-forge scrapy

yabberth
  • 507
  • 4
  • 9
  • Thanks yabberth, I did that and it worked. But I am following a tutorial for scrapy where they do stuff in the terminal (assuming this is mac command prompt). But when I try typing scrapy into the anaconda command prompt I get "Fatal error in launcher unable to create process using '"d:\bld\scrapy_1572360424769\_h_env\python.exe" "C:\ProgramData\Anaconda3\Scripts\scrapy.exe" ' (base) C:\WINDOWS\system32> – Christopher Ell Dec 07 '19 at 13:05
  • Can you link the tutorial please? – yabberth Dec 07 '19 at 13:19
  • It is https://www.datacamp.com/community/tutorials/making-web-crawlers-scrapy-python#shell under the heading scrapy shell – Christopher Ell Dec 08 '19 at 10:10
1

I finally have got scrapy installed on my computer and below I outline the things I had to fix and tried for anyone else having similar issues. Please feel free to correct me or inform me of any redundant steps I may include

Thanks to yabberth for answering my question his method now works on my computer, but there were a few other things I had to fix up first to get things working (many of which seem obvious now in hindsight).

Environmental Variables

My environmental variables were not set up properly for a few reasons.

to access environmental variables I just search for "View Advanced System settings" in windows 10, I then click on Environmental variables. Then I see "User Variables" and "System Variables" both of which have a field called path. path is the field I will be modifying so I double click it on both and am presented with a list of paths.

  1. In both I include the path to where my Python is currently installed, which unlike most people for some reason is under C:\Users\MyUsername\AppData\Local\Programs\Python\Python37\ as well as: C:\Users\cell\AppData\Local\Programs\Python\Python37\Scripts\

But it still wasn't working so:

  1. I currently have Python 3.7 installed, but previously had 3.6 and for some reason the paths for 3.6 were still there (not sure if the uninstall is meant to get rid of them or I was meant to do that). Once I deleted these pip started working again and I could install scrapy

After completing the above two steps I was able to install scrapy using windows command prompt using:

pip install scrapy

or as yabberth suggested on anaconda with 

conda install -c conda-forge scrapy
Christopher Ell
  • 1,878
  • 3
  • 16
  • 25