Python time.sleep command in a loop with if/else condition

Question

I have a question about the time.sleepcommand in a if/elseloop. I make an api request for different places with the latitude and longitude. All in all, I need the request for 5 years. I use two forloops over the latitude and longitude and over the requested five years. The problem is that I can only send 6 requests per minute. Because of that, I use the time.sleepcommand as follow:

 for lat, lon in zip(lats, lons):
     for year in years:
         args = {
             'interpolate': False,
             'lat': lat,
             'lon': lon,
             'date_from': year + '-01-01',
             'date_to': year + '-12-31',
             'capacity': 500,
             'height': 44,
             'turbine': 'Enercon E40 500',
             'format': 'json',
             'metadata': False,
             'raw': True,
         }
         r = s.get(url, params=args)
         data = pd.read_json(r.text, orient='index')
         df = df.append(data)
     time.sleep(61)

The function works good and the time.sleepcommand too! But in this form, the function waits 61 seconds after the last passing. I wanna make this a litte bit "smoother" with an if/elseloop in that way, if the loop uses the last latitude/longitude, the time.sleepcommand can be ignored. The years are a normal list and the latitude/longitude are in a seperate excel table. Has someone a idea, how I can write this? The ways I tried it doesn't work and I need some new inspiration...Thank you!

@StephenRauch good qustion, I didn't thought about it...Can you explain me, how do you mean it? — Pyrmon55, May 06 '18 at 18:50
Everytime you do a request keep track of the time in a fifo: https://stackoverflow.com/questions/1931589/python-datatype-for-a-fixed-length-fifo. Then you can know how long it has been since the last 6 requests. — Stephen Rauch, May 06 '18 at 18:52
Ok I see what you mean. I'm not really sure how to write this, so I think I try to use the way with if/else...Have you an idea for that way, how it could be possible to write this? — Pyrmon55, May 06 '18 at 19:08

score 1 · Accepted Answer · answered May 06 '18 at 19:22

The 6 reuqests per minute doesn't match the 5 requests per inner loop, so you need to decouple the sleep from the loop. You don't want a gratuitous sleep when it turns out you are at the end of the iteration and sleeping for no good reason. And you don't want a hard coded sleep time because that doesn't account for the time you spent processing.

A solution is to write a class that tracks the time constraints for you. It can remember the time and once every six times it is called, wait for the delta time between 60 seconds and whatever time has already passed.

Place the pacer at the start of the inner loop and it will sleep once every 6 requests when you already know that you've got another request pending.

class Pacer:

    def __init__(self, burst_size, seconds):
        self.tick = time.time()
        self.burst_size = burst_size
        self.seconds = seconds
        self.count = 1

    def step(self):
        if self.count % self.seconds == 0:
            delta = self.seconds - (time.time() - self.tick)
            if delta > 0:
                time.sleep(delta)
            self.tick = time.time()
        self.count += 1

pacer = Pacer(6, 60)
for lat, lon in zip(lats, lons):
    for year in years:
        pacer.step()
        args = {
            'interpolate': False,
            'lat': lat,
            'lon': lon,
            'date_from': year + '-01-01',
            'date_to': year + '-12-31',
            'capacity': 500,
            'height': 44,
            'turbine': 'Enercon E40 500',
            'format': 'json',
            'metadata': False,
            'raw': True,
        }
        r = s.get(url, params=args)
        data = pd.read_json(r.text, orient='index')
        df = df.append(data)

Python time.sleep command in a loop with if/else condition

1 Answers1