75

I have written a program which sends more than 15 queries to Google in each iteration, total iterations is about 50. For testing I have to run this program several times. However, by doing that, after several times, Google blocks me. is there any ways so I can fool google maybe by adding delays between each iteration? Also I have heard that google can actually learn the timesteps. so I need these delays to be random so google cannot find a patter from it to learn my behavior. also it should be short so the whole process doesn't take so much. Does anyone know something, or can provide me a piece of code in python?

feetwet
  • 3,248
  • 7
  • 46
  • 84
Hossein
  • 40,161
  • 57
  • 141
  • 175
  • 3
    What are you doing that you need to slam google with more than 750 requests in a short timeframe? Is there any way you can batch up the requests so as to send fewer requests? – Benn Oct 29 '10 at 17:16
  • No, I have developed a QA system. and I need the results incrementally, that's why I have to send the queries one by one. – Hossein Oct 29 '10 at 17:19
  • I know, see the below answer please ;) – Hossein Oct 29 '10 at 17:27
  • 8
    If you are testing some other system, you might be better off to create a mock-google that will respond with the results that your SUT expects. – bstpierre Oct 29 '10 at 17:43

5 Answers5

169

First, Google probably are blocking you because they don't like it when you take too many of their resources. The best way to fix this is to slow it down, not delay randomly. Stick a 1 second wait after every request and you'll probably stop having problems.

That said:

from random import randint
from time import sleep

sleep(randint(10,100))

will sleep a random number of seconds (between 10 and 100).

nmichaels
  • 49,466
  • 12
  • 107
  • 135
  • 5
    A couple of thoughts, inserting a 1 second wait after each query will increase runtime by a little more than 12 minutes. Inserting an average of a 50 second wait after each query will yield an increase of more than 10 hours. This might be onerous, even for testing. – jball Oct 29 '10 at 17:33
  • 3
    jball: True, and `sleep` can take floats, so the numbers can be tuned. Personally, I like the other answers better as solutions to this particular problem but this answers the question for people who might want to insert random delays into something in the future. – nmichaels Jan 14 '14 at 17:14
  • Can we make it 0.5 seconds lol yesterday I didn't add a delay and I was blocked from using google for a moment. – Hani Goc May 07 '15 at 13:44
  • 7
    I've used `random.random.uniform(1, 3)` and google seem to be satisfied – klapshin Mar 20 '19 at 15:56
7

Best to use:

from numpy import random
from time import sleep

sleeptime = random.uniform(2, 4)
print("sleeping for:", sleeptime, "seconds")
sleep(sleeptime)
print("sleeping is over")

as a start and slowly decreasy range to see what works best (fastest).

Hrvoje
  • 13,566
  • 7
  • 90
  • 104
4

Since you're not testing Google's speed, figure out some way to simulate it when doing your testing (as @bstpierre suggested in his comment). This should solve your problem and factor its variable response times out at the same time.

martineau
  • 119,623
  • 25
  • 170
  • 301
2

For anyone stumbling here for the general "how to add random delay to my routine" case in 2022, numpy's recommended method [1] is to use their random number generator class:

from numpy.random import default_rng
from time import sleep

rng = default_rng()

# generates a scalar [single] value greater than or equal to 1
# but less than 3
time_to_sleep = rng.uniform(1, 3)

sleep(time_to_sleep)

[1] https://numpy.org/doc/stable/reference/random/index.html#quick-start

Justin
  • 500
  • 8
  • 18
1

Also you can try to use few proxy servers for prevent ban by IP adress. urllib support proxies by special constructor parameter, httplib can use proxy too

seriyPS
  • 6,817
  • 2
  • 25
  • 16