how to start scrapyd server on EC2 instance

Question

I have setup an instance on aws. Now I want to start scrapyd on a particular port. according to documentation

aptitude install scrapyd-X.YY

but aptitude is not found. I have tried to installing aptitude using yum but there is no match found (may be it only works with apt-get, but I have yum ap-get is also missing)

can any one please help me that is there any other way to do this ??

yes, amazon's OS uses yum, see below on instructions for using yum to install pip, then pip for installing scrapyd — Tom Swifty, Oct 31 '13 at 20:02

Tom Swifty · Answer 1 · 2013-10-31T19:17:42.507

2

If you first install pip:

sudo yum install python-pip

you can use pip to install scrapyd like so

pip install scrapyd

source: http://scrapyd.readthedocs.org/en/latest/install.html

edited Oct 31 '13 at 19:17

answered Oct 31 '13 at 19:12

Tom Swifty

2,864
2
16
25

yes it works, but as documentation shows there is not config file created at /etc/scrapyd – Tasawer Nawaz Oct 31 '13 at 20:04

Gaia · Accepted Answer · 2013-11-03T16:53:44.887

1

You are using an yum based OS, not an apt based OS. Forget any commands that involve apt or a variation thereof.

Skip the steps you've already done:

yum install python
yum install python-pip
yum install libxml2-python
pip install Scrapy

As for libxml2-python, keep in mind that "versions prior to 2.6.28 are known to have problems parsing certain malformed HTML, and have also been reported to contain leaks, so 2.6.28 or above is highly recommended"

edited Nov 03 '13 at 16:53

answered Oct 31 '13 at 19:20

Gaia

2,872
1
41
59

thanx, I have already install python and scrapy I just want to start scrapyd process on particular port like.. ip:6811 etc so that I can deploy my project there – Tasawer Nawaz Oct 31 '13 at 20:04

how to start scrapyd server on EC2 instance

2 Answers2