How to get response status code on process_exception in Scrapy?

Question

I want to retry Scrapy request when if it gets an exception and the response status code is 429. The problem is I don't know how to get the response status on the process_exception. How can I do it since it seems there is no way to access response object inside process_exception?

class FakeUserAgentErrorRetryMiddleware(RetryMiddleware):

    def process_exception(self, request, exception, spider):
        if type(exception) == FakeUserAgentError: return self._retry(request, exception, spider)

score 0 · Answer 1 · answered Dec 21 '17 at 14:20

0

If you want to retry a request when you have response.status = 429 you should add 429 in RETRY_HTTP_CODES

RETRY_HTTP_CODES = [429]

You can set also set the maximum number of times to retry with RETRY_TIMES

answered Dec 21 '17 at 14:20

parik

2,313
12
39
67

That doesn't work on Scrapy Splash, so I wanted to try this way – Aminah Nuraini Dec 21 '17 at 14:22
1

You didn't mention that you use Splash :) – parik Dec 21 '17 at 14:24
I mention it now – Aminah Nuraini Dec 21 '17 at 14:25
1

it's better to mention it in your question for getting right answer faster :) – parik Dec 21 '17 at 14:25

How to get response status code on process_exception in Scrapy?

1 Answers1