Asyncio stdout - failing

Question

I'm trying to get to grips with using asyncio to speed up a process of calling an external tool to analyse multiple audio files. I'm working on windows, python 3.6 (anaconda installation) and the problem I'm having here is that the calls don't seem to wait or the results are never received through stdout.

Any ideas?

import os
import asyncio

external_tool = r"C:\path\to\external\tool.exe"

def filepaths_generator(root):
    '''
    Simple generator to yield filepaths
    '''
    for path, dirs, files in os.walk(root):
        for f in files:
            yield os.path.join(path,f)


async def async_subprocess_command(*args):
    '''
    A function to run the asyncio subprocess call
    '''
    # Create subprocess
    process = asyncio.create_subprocess_exec(
        *args,
        # stdout must a pipe to be accessible as process.stdout
        stdout=asyncio.subprocess.PIPE)
    # Wait for the subprocess to finish
    stdout, stderr = await process.communicate()
    # Return stdout
    return stdout.decode().strip()

async def add_external_tool_data_to_dict(filepath):
    external_tool_data = await async_subprocess_command(external_tool,filepath)
    metadata_dict[filepath] = external_tool_data
    return

metadata_dict = {}

loop = asyncio.get_event_loop()
#loop = asyncio.ProactorEventLoop() - Tried this, it doesn't help!

filepaths = filepaths_generator(r"C:\root\path")
tasks = []

for filepath in filepaths:
    #Create tasks in for the loop and add to a list
    task = loop.create_task(add_external_tool_data_to_dict(filepath))
    tasks.append(task)
#Run the tasks
loop.run_until_complete(asyncio.wait(tasks))

loop.close()
print(metadata_dict)

If I use the loop = asyncio.ProactorEventLoop() I get the error: — user3535074, Aug 15 '17 at 19:19
File "C:\Users\danie\Anaconda3\lib\asyncio\base_events.py", line 340, in _make_subprocess_transport raise NotImplementedError NotImplementedError — user3535074, Aug 15 '17 at 19:20
After you tried the `loop = asyncio.ProactorEventLoop()`, did follow up with: `asyncio.set_event_loop(loop)` ? — Gerrat, Aug 15 '17 at 20:00

score 6 · Accepted Answer · answered Aug 15 '17 at 20:07

For starters there's this:

On Windows, the default event loop is SelectorEventLoop which does not support subprocesses. ProactorEventLoop should be used instead

Example to use it on Windows:

import asyncio, sys

if sys.platform == 'win32':
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)

I would start by switching to this alternate event loop.

score 1 · Answer 2 · answered Aug 15 '17 at 20:07

1

As per Gerrat's comment, the solution was to use the ProactorEventLoop and add an additional line thereafter:

loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)

answered Aug 15 '17 at 20:07

user3535074

1,268
8
26
48

Asyncio stdout - failing

2 Answers2