where's data store after deploy spider scrapyd python?

Question

i deployed and scheduled my spider on http://localhost:6800/ success, but where's data of item store ? How could i get them ?

thanks so much !

Do you have any pipelines configured? Show your `settings.py`. — alecxe, Nov 23 '14 at 01:10
Yes, i have a pipiline, if i using command `scrapy crawl myspider`, i run ok and stored data in my json file, but when i deploy my project with comand `curl http://localhost:6800/schedule.json -d project=project_name -d spider=myspider`, it deploy sucess but i can't not find my data.This is my setting .py : `BOT_NAME = 'comicvn2' SPIDER_MODULES = ['comicvn2.spiders'] NEWSPIDER_MODULE = 'comicvn2.spiders' ITEM_PIPELINES = { 'comicvn2.pipelines.Comicvn2Pipeline' }` — tuancoi, Nov 23 '14 at 09:34

score 1 · Accepted Answer · edited Dec 03 '14 at 01:45

1

You should handle it in process_item() instead of handle by scrapyd.Because scrapyd just logs items, there is no difference in a way to store data except log file. You can find it in documentation of scrapyd.

edited Dec 03 '14 at 01:45

answered Dec 03 '14 at 01:37

Ngo Van Anh Tan

26
2

thanks your help, in process_item(),I got and saved data in json type. It resolve my prolem. – tuancoi Dec 09 '14 at 04:28

score 0 · Answer 2 · answered Nov 25 '14 at 11:19

0

In scrapyd you have to define item directory in scrapyd.conf file, for more information you can look at scrapyd documentation scrapyd.conf

answered Nov 25 '14 at 11:19

Tasawer Nawaz

927
8
19

where's data store after deploy spider scrapyd python?

2 Answers2