2

I just installed scrapyd on Ubuntu 14.04 and after the installation, I simply typed "scrapyd" into the command line and got:

Failed to load application: No module named txweb

I used both the general and Ubuntu specific installations and neither seem to work. Just for some background, I decided to use scrapyd after having created a project that ran multiple spiders but was not able to pipeline the scraped data into two different tables. I looked at some other SO posts like this one, but they seem to be addressing older versions of scrapyd which probably won't be relevant anymore.

Can someone show me how to get started with scrapyd, because the information provided in the docs doesn't seem to be working for me, thanks.

Community
  • 1
  • 1
loremIpsum1771
  • 2,497
  • 5
  • 40
  • 87

1 Answers1

4

although the docs has apt-get ... pip is the better way to install scrapy (Never failed for me)

sudo pip install scrapyd works for me and then scrapyd doesn't give any errors.

When you install with apt-get - twisted may not be getting installed, but pip takes care of that.

Try uninstalling your apt-get scrapyd first, and then use pip to install it

sudo apt-get purge scrapyd
sudo pip install scrapyd

-------------- UPDATE ------------

Answering your next question in the comment ...

If you want to run scrapyd-deploy you need to also install the package scrapy-client which is not installed along with scrapy. This is mentioned in the docs - http://scrapyd.readthedocs.org/en/latest/deploy.html

You can install it using sudo pip install scrapyd-client

AbdealiLoKo
  • 3,261
  • 2
  • 20
  • 36
  • When I just type in "scrapyd", that seems to be working, but when I use: "scrapyd-deploy" I get: "command not found." Is there some command I should run before trying to deploy or something? Thanks for the answer btw. – loremIpsum1771 Jul 13 '15 at 22:47