Organize Python function return into a clean for loop and check if time exceeded

Question

I have a function that I run on a number of data frames. I'd like to be able to organize this a bit better and add in a timeout statement. I'm a bit new to this...

organize data frames -

d = {}
dfs = [1,2,3]
for name in dfs:
   df[name] = some_function()

And then set-up to organize these into a clean loop that checks how long the df takes to run. So it would run df_1 and df_2 because they take 5 seconds but would skip and print df_3 that it took x number of seconds.

def timeout():
   # check how long df_1, df_2, df_3 takes and if takes longer than 30 seconds then print out the df name

Alain T. · Answer 1 · 2021-03-16T15:48:02.133

You could use a Timer (from the threading module) but your loop must cooperate and stop then the time is expired. This could also be done by checking the elapsed time at each iteration but I believe the Timer approach would incur less overhead.

Assuming you are using an iterator for the loop, you can define a generic "timeOut" function to force it to stop after a given number of seconds:

from threading import Timer

def timeout(maxTime,iterator):
    stop = False
    def timedOut(): # function called when timer expires
        nonlocal stop
        stop = True
    t = Timer(maxTime,timedOut)
    t.start()
    for r in iterator:
        if stop: break  # you could print the content of r here if needed.
        yield r
    t.cancel()

output:

for i in timeout(3,range(1000)):
    for _ in range(10000000): pass
    print(i)

1
2
3
4
5
6
7
8     # it stopped here after 3 seconds with 992 iterations to go

In your example, this could be:

d = {}
dfs = [1,2,3]
for name in timeout(5,dfs):
   df[name] = some_function()

Note that this will stop the loop on dfs when the total processing time exceeds 5 seconds but it cannot interrupt what is going on inside some_funtion() if it exceeds the total time.

If you need a timeout that is not tied to a specific loop, you can create an instance of a Timer that you store in global variable or in a singleton class and check its state at appropriate places in you code:

t = Timer(25,lambda:pass) # timer will not do anything when expired
t.start()                 # but will no longer be alive after 25 seconds

...
    # at appropriate places in your code
    if not t.is_alive(): return  # or break or continue ...

Organize Python function return into a clean for loop and check if time exceeded

1 Answers1