0

site behaves noramlly when accessed through browser but the redirection issue occurs while accessing the site through scrapy bots. I use Scrapy-Crawlera proxy services, still site redirects endlessly. If i use handle_httpstatus_list = [302] or dont_redirect = True, i will get the response as

Object moved here.

I am hoping for any suggestions or answers as to why this behaviour occurs on certain sites? Unfortunatly i can not share the website URL.

Shashikiran Neelakantaiah
  • 77
  • 6
  • Providing a code example might be helpful. I assume you are already using an appropriate user agent? – carpa_jo Apr 29 '20 at 08:43
  • unfortunatly i can not provide the code example and yes the proxy service will handle the user agent appropriatly – Shashikiran Neelakantaiah Apr 29 '20 at 09:49
  • Why? It can be an antibot measure, the time a bot is crawling endless fictional redirects is time it is not extracting actual data, and it may discourage unexperienced developers. It can also be a server-side error (when the same happens in a web browser). – Gallaecio May 01 '20 at 15:54
  • It does not happen in a web browser, its when running the url through scrapy, but sometimes randomnly a successfull request will happen, but around 90% of the time, it redirects and the script stops running as the max-redirection is reached. – Shashikiran Neelakantaiah May 26 '20 at 04:24

0 Answers0