Scrapy on AWS EC2 : where to write the items?

Question

I have a working spider on my local machine, which writes items to a local postgres database.

I am now trying to run the same spider through scrapyd on an EC2 instance. This obviously won't work, because the code (models, pipelines, settings files) refers to a database on my local machine.

Which adaptations should I implement to make this work ?

score 1 · Accepted Answer · answered Nov 19 '14 at 03:22

1

Found it, answer was easier than i thought. In the settings.py file, delete the settings for ITEM_PIPELINES and DATABASE. After deletion, deploy the project through scrapyd on EC2.

By default, items will now be written as JSON-lines. This can be overridden with FEED_FORMAT and FEED_URI :

sudo curl http:/xxxxxxxxx.us-west-2.compute.amazonaws.com:6800/schedule.json -d project=xxxxxxxxxx -d spider=xxxxxxxxx -d setting=FEED_URI=/var/lib/scrapyd/items/xxxxxxxxxx.csv -d setting=FEED_FORMAT=csv

answered Nov 19 '14 at 03:22

S Leon

331
1
4
18

1

you can also you the RDS service if you still want to store the items to a DB. – Nov 20 '14 at 02:35
Did exactly that now, using a Postgres instance on RDS. And for free ! – S Leon Nov 20 '14 at 04:15

Scrapy on AWS EC2 : where to write the items?

1 Answers1