8

I am using the following function to force a coroutine to run synchronously:

import asyncio
import inspect
import types
from asyncio import BaseEventLoop
from concurrent import futures


def await_sync(coro: types.CoroutineType, timeout_s: int=None):
    """

    :param coro: a coroutine or lambda loop: coroutine(loop)
    :param timeout_s:
    :return:
    """
    loop = asyncio.new_event_loop()  # type: BaseEventLoop
    if not is_awaitable(coro):
        coro = coro(loop)
    if timeout_s is None:
        fut = asyncio.ensure_future(coro, loop=loop)
    else:
        fut = asyncio.ensure_future(asyncio.wait_for(coro, timeout=timeout_s, loop=loop), loop=loop)
    loop.run_until_complete(fut)
    return fut.result()

def is_awaitable(coro_or_future):
    if isinstance(coro_or_future, futures.Future):
        return coro_or_future
    elif asyncio.coroutines.iscoroutine(coro_or_future):
        return True
    elif asyncio.compat.PY35 and inspect.isawaitable(coro_or_future):
        return True
    else:
        return False

However, intermittently, it will freeze upon simply trying to create a new loop: loop = asyncio.new_event_loop(). Inspecting the stack traces shows me the exact location where it hangs:

File: "/src\system\utils.py", line 34, in await_sync
  loop = asyncio.new_event_loop()  # type: BaseEventLoop
File: "\lib\asyncio\events.py", line 636, in new_event_loop
  return get_event_loop_policy().new_event_loop()
File: "\lib\asyncio\events.py", line 587, in new_event_loop
  return self._loop_factory()
File: "\lib\asyncio\selector_events.py", line 55, in __init__
  self._make_self_pipe()
File: "\lib\asyncio\selector_events.py", line 116, in _make_self_pipe
  self._ssock, self._csock = self._socketpair()
File: "\lib\asyncio\windows_events.py", line 295, in _socketpair
  return windows_utils.socketpair()
File: "\lib\socket.py", line 515, in socketpair
  ssock, _ = lsock.accept()
File: "\lib\socket.py", line 195, in accept
  fd, addr = self._accept()

What can be causing such an issue, in a library as low level as socket? Am I doing something wrong? I am using Python 3.5.1.

Edit: I filed a bug report here but Guido recommended me to continue seeking help on StackOverflow.

Erwin Mayer
  • 18,076
  • 9
  • 88
  • 126
  • Possible duplicate of [Event loop created by asyncio.new\_event\_loop hangs](http://stackoverflow.com/questions/34470856/event-loop-created-by-asyncio-new-event-loop-hangs) – kwarunek Mar 08 '16 at 20:16
  • This is not a duplicate at all. My code hangs upon the simple calling of asyncio.new_event_loop, the other question talks about a loop that hangs AFTER it has been created. – Erwin Mayer Mar 09 '16 at 07:41
  • Little note: inside is_awaitable you're checking for isinstance of `futures.Future`. If I'm right, it's wrong: you should check for `asyncio.Future`. These classes are different. – Mikhail Gerasimov Mar 13 '16 at 02:15
  • About question: did you try to close your loop and sockets you use? `rsock.close()`, `wsock.close()`, `loop.close()` – Mikhail Gerasimov Mar 13 '16 at 02:26
  • @germn is_awaitable is mostly a copy paste from the code inside asyncio.ensure_future, to return false instead of raising an exception when a non awaitable is returned, so I hope it is correct. – Erwin Mayer Mar 13 '16 at 07:58
  • I don't have direct control over the sockets (they are not part of my code), however, as discussed with Guido in my bug report, I am indeed trying now to always close the loop explicitly. So far so good, if the problem completely disappears (not sure yet), I'll update my question with this as an answer. – Erwin Mayer Mar 13 '16 at 08:00
  • @ErwinMayer, about Future: in asyncio inside asyncio.ensure_future uses `asyncio.futures.Future` class. So if in your code uses `from asyncio import futures`, your code is correct. But if you used `from concurrent import futures` it's mistake. – Mikhail Gerasimov Mar 13 '16 at 13:31
  • You're right, I did not include the import statement in my code (which as from concurrent import futures). I just updated it to avoid any confusion! – Erwin Mayer Mar 13 '16 at 16:57
  • have you read documentation? especially: https://docs.python.org/3/library/asyncio-eventloops.html#windows imho it looks like you are trying to use SelectorEventLoop on windows, which will fail on creating pipes, maybe use different event loop implementation? – Jerzyk Jul 01 '16 at 09:09
  • @Jerzyk interesting, it is on Windows indeed (not tested on Linux). Where you do see that I am trying to use SelectorEventLoop? Fyi I was able to fix the issue by explicitly closing the loop after each use. – Erwin Mayer Jul 01 '16 at 16:41
  • 1
    @ErwinMayer: `File: "\lib\asyncio\selector_events.py", line 116, in _make_self_pipe` – Jerzyk Jul 01 '16 at 17:15
  • @ErwinMayer plus, I would try with `get_event_loop` instead of `new_event_loop` as there will be less hassle to additionally mark loop as active for the thread... – Jerzyk Jul 01 '16 at 17:18
  • @Jerzyk thanks. The issue with using `get_event_loop` is that the loop may already be used by another thread (a loop is not necessarily tied to a single thread). Here the idea is just to run async methods synchronously asap. But I think your remark on SelectorEventLoop could well explained the freezing behavior, while my method to close each loop actively after each use rather than relying on the garbage collector reliably prevents this excess socket usage. – Erwin Mayer Jul 05 '16 at 04:37
  • @ErwinMayer that's why I suggested to use `get_event_loop` instead – Jerzyk Jul 05 '16 at 12:15
  • 1
    It doesn't freeze on python 3.7.0 and also work on 3.6.1. – cglacet Feb 22 '19 at 22:18
  • I'm still trying to understand why is this function so complex, can you elaborate on your objectives? For example what is `asyncio.compat.PY35` and why do you need that kind of check? – cglacet Feb 22 '19 at 22:30
  • @ChristianG. This was an old issue (I don't remember the objectives except it was to provide a reproducible snippet), if it works well with latest versions of Python we should be all set! – Erwin Mayer Feb 23 '19 at 11:31
  • Yeah I couldn’t get my hands on 3.5.1 and I wasn’t sure it was worth going further back than 3.6. But to be honest I didn’t notice that the original question was almost 3 years old. I would still be curious to know what was causing this to happen. – cglacet Feb 23 '19 at 11:44

1 Answers1

1

I'm trying to understand what you are trying to do.

If I get it correctly you want a function that will return the same thing whether the input is a coroutine or a simple function call. If I'm correct, then this seems to work fine. 

import asyncio
import time

def await_sync(coro, timeout=None):
  if asyncio.iscoroutine(coro):
    f = asyncio.wait_for(coro, timeout)
    loop = asyncio.get_event_loop()
    return loop.run_until_complete(f)
  return coro

async def async_test(x):
    print("x", end="")
    await asyncio.sleep(x)
    print("y", end="")
    return x

def sync_test(x):
    print("x", end="")
    time.sleep(x)
    print("y", end="")
    return x

print(await_sync(sync_test(2)))
print(await_sync(async_test(3)))

This outputs the following (I guess expected) result:

xy2
xy3
cglacet
  • 8,873
  • 4
  • 45
  • 60