1

The website i scapped blocked me out by showing 406 Not Acceptable on the browser. It might i mistakenly sent too many requests at once on phython code.

So i put time.sleep(10) for each loop to not make it look like a DDoS attack, and it seems worked out.

My questions are:

  1. How long would it be reasonable to send between each request? Sleep 10 seconds for each loop makes my code running too slow.

  2. How to fix the 406 Not Acceptable error on my browsers? They still block me out, only if i chance my ip address but it's not permanent solution.

Thank you all for your answers and comments. Good day!

laph
  • 2,925
  • 2
  • 18
  • 18

1 Answers1

1

Any rate-limit errors are all subject to which website you choose to scrape / interact with. I could set up a website that only allows you to view it once per day, before throwing HTTP errors at your screen. So to answer your first question, there is no definitive answer. You must test for yourself and see what's the fastest speed you can go, without getting blocked.

However, there is a workaround. If you use proxies, then it's almost impossible to detect and stop the requests from executing, and therefore you will not be hit by any HTTP errors. HOWEVER, JUST BECAUSE YOU CAN, DOESN'T MEAN THAT YOU SHOULD- I am a programmer, not a lawyer. I'm sure there's a rule somewhere that says that spamming a page, even after it tells you to stop, is illegal.

Your second question isn't exactly related to programming, but I will answer it anyways- try clearing your cookies or refreshing your IP (try using a VPN or such). Other than changing your IP or cookies, there's not many more ways that a page can fingerprint you (in order to block you).

Xiddoc
  • 3,369
  • 3
  • 11
  • 37
  • Thanks for your answer. I meant i used a VPN to change the IP address and it works just fine. But when i disable VPN, it blocks me out again, on every browser. I did clear all cookies but still problem. It seems they block my orignal IP address – laph Mar 30 '21 at 10:19
  • @laph Yes, that seems like the problem. Try staying on a VPN for a day or 2 and maybe they'll remove the block after that. Also, if my answer is valid, then make sure to mark it as such by clicking the checkmark on the left of it. Thanks, and good luck! – Xiddoc Mar 30 '21 at 10:24