I want my scraper to stop redirect and try the same URL again. How can I do that?
I have tried meta={'dont_redirect': True}. But it ignores the request to current URL. I want the HTML content of the current URL.
Asked
Active
Viewed 128 times
1
Sabeeh Zia
- 13
- 4
-
Does this answer your question? [scrapy- how to stop Redirect (302)](https://stackoverflow.com/questions/15476587/scrapy-how-to-stop-redirect-302) – Andrejs Cainikovs Aug 24 '20 at 12:44
-
When I set meta={'dont_redirect': True} it stop redirecting but also does not get the hml content. I want the HTML content – Sabeeh Zia Aug 24 '20 at 13:21
-
create minimal working code with real URL which makes problem. And then we can test it to see where is the problem. – furas Aug 24 '20 at 16:20
-
Did you also set `httpstatus_list` as explained in https://stackoverflow.com/a/15477884/939364 ? – Gallaecio Aug 25 '20 at 14:44
-
Any results on that? I have the same problem where the url gets redirected. When I handle it with the httpstatus_list some attributes do not get parsed although they exist on the page. – Blind0ne Oct 24 '21 at 14:00