0
import requests
from requests_ip_rotator import ApiGateway, EXTRA_REGIONS

if __name__ == "__main__":

    # Create gateway object and initialise in AWS
    gateway = ApiGateway("https://spare.avspart.com", regions=EXTRA_REGIONS, access_key_id = 'my key', access_key_secret = 'my secret key')
    gateway.start(force=True)

    # Execute from random IP
    session = requests.Session()
    # session.max_redirects = 100
    session.mount("https://spare.avspart.com", gateway)

    # setting User-Agent header
    session.headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36'
    response = session.get("https://spare.avspart.com/catalog/case/64848/4534337/677993/")
    print(response.status_code)

    # Delete gateways
    gateway.shutdown()

I am trying to scrape this page"https://spare.avspart.com/catalog/case/64848/4534337/677993/" using requests-ip-rotator because I was blocked using requests.get() but when I try and access it I get TooManyRedirects: Exceeded 30 redirects. error.

I have read through most of the posts on this problem and tried various things such as changing the session.max_redirects and trying different types of headers, and also reaching out to the library creator. This answer Accepted answer for the same issue seems to solve the problem, but when I try and implement this in my code the issue persists.

It would be great if anyone has any recommendations for other things I can try.

  • 1
    A simple `requests.get("https://spare.avspart.com/catalog/case/64848/4534337/677993/")` works for me. Perhaps you have made too many requests? Are you behind any kind of firewall? Try using a VPN? – Martin Evans Nov 10 '21 at 18:00
  • @Martin Evans Actually I manged to block my IP address by making too many requests so I am using the library ```requests_ip_rotator``` I guess that's where the issue is. –  Nov 10 '21 at 18:13

0 Answers0