Event loop in asyncio is overflowing. Though adding 4 executions at a time it is overloading

Question

In the following code I am calling getSUEPEvent() funtion 4 time for single loop. I restart loop again for next 4. Still execution keeps on adding to loop. If loop is global than can any one suggest another strategy to can funtion n number of time by grouping or any else means.

def SEUPCustomers(featurecode,threshholdTime):
    # headers = buildHeaders()
    with open("ActiveCustomers.csv","r") as f:
        SEUPCustomersList = []
        csvReader = csv.reader(f)
        tasks = []
        for row in csvReader:
            tasks.append(asyncio.ensure_future(getSEUPEvents(featurecode,row,threshholdTime,SEUPCustomersList)))

        for task in range(0,len(tasks),4):
            loop = asyncio.get_event_loop()
            loop.run_until_complete(asyncio.wait(tasks[task:task+4]))
            loop.close()

score 3 · Accepted Answer · answered Feb 01 '19 at 09:39

ensure_future and run_until_complete don't work the way you expect them to. Here is what they do:

ensure_future schedules the awaitable to run in the main loop, effectively creating what could be called "background task", which will run whenever the main loop runs;
run_until_complete submits the given awaitable to the event loop and runs the event loop until that particular future completes.

So, if you submit 100 tasks to the event loop and then use run_until_complete to wait until one of them finishes, the loop will run all 100 tasks, and stop once the one whose completion you're interested in finishes.

To write the code you wanted, you can simply avoid the ensure_future step:

def SEUPCustomers(featurecode,threshholdTime):
    # headers = buildHeaders()
    with open("ActiveCustomers.csv","r") as f:
        SEUPCustomersList = []
        csvReader = csv.reader(f)
        coros = []
        for row in csvReader:
            coros.append(getSEUPEvents(featurecode,row,threshholdTime,SEUPCustomersList))

        loop = asyncio.get_event_loop()
        for i in range(0,len(coros),4):
            loop.run_until_complete(asyncio.wait(coros[i:i+4]))

Also, loop.close() is incorrect if you plan to use the loop later on. If you call loop.close() at all, you should call it once you're completely done with the event loop.

Event loop in asyncio is overflowing. Though adding 4 executions at a time it is overloading

1 Answers1