requests disappear after queueing in scrapy

Question

Scrapy seems to complete without processing all the requests. I know this because i am logging before and after queueing the request and I can clearly see that.

I am logging in both parse and error callback methods and none of them got called for those missing requests.

How can I debug what happened to those requests?

did you add `dont_filter=True` when re-queueing the request? — Tarun Lalwani, Sep 28 '17 at 10:15
@TarunLalwani Bingo! it worked but I am not sure why would scrapy drop it because in the entire 50 URLs, that URL had no duplicate. — comiventor, Sep 28 '17 at 10:23

score 0 · Accepted Answer · answered Sep 28 '17 at 10:27

0

You need to add dont_filter=True when re-queueing the request. Though the request may not match other request but Scrapy remembers what requests it has already made and it will filter out if you re-queue it. It will assume it was by mistake.

answered Sep 28 '17 at 10:27

Tarun Lalwani

142,312
9
204
265

requests disappear after queueing in scrapy

1 Answers1

Linked