How would one go about implementing a live feedback on what the spider is doing ? I have the logs that tell me each second pages per second and items, but i would like to access that via a webservice.
Maybe I'm wrong, but I found in the scrapy docs that there is something that should do this http://localhost:6080/stats/spider_stats but it returns empty.
Any ideas are welcome.
I would like to be able to do something like curl http://localhost:6800/stats.json -d project=default -d spider=somespider
and it would return {"pages_crawled": "650","pages_per_min":"342","items_scraped":"286","items_per_min":"156"}
Thanks,
