Asyncio errors: CLI - python daemon example but has errors/undesired behavior on exit

Question

I've developed a pattern to use for a commanding a python daemon through cmd module shell using an eventloop. However, its not ready yet because I can't figure out how to gracefully exit the two applications (I'm still learning asyncio and can't figure out the following problem). When the cmd module is commanded to exit I get:

(Cmd) exit
Shutting down client...
Traceback (most recent call last):
  File "client_loop.py", line 10, in <module>
    loop.run_until_complete(test.cmdloop())
  File "...\asyncio\base_events.py", line 467, in run_until_complete
    future = tasks.ensure_future(future, loop=self)
  File "...\lib\asyncio\tasks.py", line 526, in ensure_future
    raise TypeError('An asyncio.Future, a coroutine or an awaitable is '
TypeError: An asyncio.Future, a coroutine or an awaitable is required

I'm not good with asyncio yet, what am I doing wrong? Sorry for the long question but the files/errors make it long and hopefully makes it easier to debug.

Here are the supporting files:

shell.py

# implementation of the (Cmd) prompt with history functionality
# standard imports
import cmd as cmd

class Shell(cmd.Cmd):
    def __init__(self, **kwargs):
        cmd.Cmd.__init__(self, **kwargs)
        self.eventloop = None
        self.shutdown_client = None
        self.tcp_echo_client = None
    
    def set_eventloop(self, loop):
        self.eventloop = loop
    
    def set_funcs(self, tcp_echo_client, shutdown_client):
        self.tcp_echo_client = tcp_echo_client
        self.shutdown_client = shutdown_client

    def do_exit(self,*args):
        """
        Exits the shell gracefully
        :param args:
        :return:
        """
        print('Shutting down client...')
        self.shutdown_client(self.eventloop)
        return True

    def default(self, line):
        try:
            self.eventloop.run_until_complete(self.tcp_echo_client(line, self.eventloop))
        except SystemExit:
            pass

server.py

# server logic to parse arguments coming over the TCP socket and echo it back

# standard imports
import asyncio

async def handle_echo(reader, writer):
    data = await reader.read(100)
    message = data.decode()
    addr = writer.get_extra_info('peername')
    print("Received %r from %r" % (message, addr))

    print("Send: %r" % message)
    writer.write(data)
    await writer.drain()

    print("Close the client socket")
    writer.close()

loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888, loop=loop)
server = loop.run_until_complete(coro)

# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
    loop.run_forever()
except KeyboardInterrupt:
    pass
finally:
    # Close the server
    for task in asyncio.Task.all_tasks():
        loop.run_until_complete(task)

    server.close()
    loop.run_until_complete(server.wait_closed())
    loop.stop()
    loop.close()
    exit(0)

client.py

# client functions to send message over TCP and process response
# standard imports
import asyncio

# user imports
import shell

async def tcp_echo_client(message, loop):
    reader, writer = await asyncio.open_connection('127.0.0.1', 8888,
                                                   loop=loop)

    print('Send: %r' % message)
    writer.write(message.encode())

    data = await reader.read(100)
    print('Received: %r' % data.decode())

    print('Close the socket')
    writer.close()

def shutdown_client(loop):
    loop.stop()

    # Find all running tasks:
    pending = asyncio.Task.all_tasks()

    # Run loop until tasks done:
    loop.run_until_complete(asyncio.gather(*pending))

loop = asyncio.get_event_loop()
test = shell.Shell()
test.set_eventloop(loop)
test.set_funcs(tcp_echo_client, shutdown_client)
loop.run_until_complete(test.cmdloop())
loop.close()

Can you shorten your code to a **minimal** example that still exhibits the asyncio issue you are asking about? It is hard to follow the layers of abstraction of your program, which are likely unrelated to the error. (The code also appears to be incomplete; you call `test.cmdloop()`, but don't seem to define `cmdloop` anywhere.) — user4815162342, May 25 '21 at 20:03
cmdloop() comes from cmd.Cmd. Its honestly as minimal as it can get. Its client/server where the client is a shell and they are sending argsparse commands to each other. This is the pattern that I am after and serves as the base. I could remove arg.py but that has little gain. Have you tried running the program yourself? You might be surprised how easy it is to pick it up. — LeanMan, May 25 '21 at 21:03
Ok I took your advice. I made it even simpler and still create the same problems. Please take a look now. Should also work if you spin up the client/server and start sending messages between one another. — LeanMan, May 25 '21 at 21:33
I really don't want to sound demanding, and perhaps someone else won't mind the format of the question, so I'm writing this just to clarify: I was referring to the fact that the example comprised four source files and abstractions like `cmd.Cmd` apparently unrelated to asyncio. The idea with providing a minimal example is that you narrow down the code to be as close as possible to where the problem lies, so that it is easier to understand and resolve it. Even after shortening, your example still has multiple source files and uses `Cmd`, so it at least doesn't *seem* minimal. — user4815162342, May 25 '21 at 22:04
The entry point seems fishy - how is it supposed to work given that `cmdloop` is not async and you're passing it to `run_until_complete()`? Also, you're running other instances of `run_until_complete()` inside it. Perhaps the underlying problem is that `cmd.Cmd` is just not compatible with asyncio? Async programs typically require the use of async libraries from the ground up, at least when calling into anything that can block. — user4815162342, May 25 '21 at 22:06
You are going to have to deal with the cmd as that is part of the question. If you believe its unrelated to the issue then please post an answer. — LeanMan, May 25 '21 at 22:10
Yeah, my last comment speculates about that. Also, I don't _have_ to deal with any particular aspect of the question, I'm just a volunteer like everyone else here. :) — user4815162342, May 25 '21 at 22:12
Point is that the solution requires to be a shell. I guess you can consider it a requirement. I cant really make that any more clear. — LeanMan, May 25 '21 at 22:16
I've been looking through my example. I think I see where I need to fix it. I don't understand what you mean by "Exit TCP sockets" — LeanMan, May 26 '21 at 00:31
Even though this wont show immediate improvement but I am going to leave this for others to read: https://insights.dice.com/2019/04/18/stack-overflow-many-jerks/ — LeanMan, May 26 '21 at 01:16

LeanMan · Answer 1 · 2021-05-26T03:34:33.187

The problem lies in the fact that cmd doesn't require asyncio to process input from the user and send a message to through the TCP/IP socket. Simply removing asyncio from the client side solved the problem. It still offers the shell implementation and the client-server pattern. Here is the new code:

server.py

# server logic to parse arguments coming over the TCP socket and echo it back

# standard imports
import asyncio

async def handle_echo(reader, writer):
    data = await reader.read(100)
    message = data.decode()
    addr = writer.get_extra_info('peername')
    print("Received %r from %r" % (message, addr))

    print("Send: %r" % message)
    writer.write(data)
    await writer.drain()

    print("Close the client socket")
    writer.close()

loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888, loop=loop)
server = loop.run_until_complete(coro)

# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
    loop.run_forever()
except KeyboardInterrupt:
    pass
finally:
    # Close the server
    for task in asyncio.Task.all_tasks():
        loop.run_until_complete(task)

    server.close()
    loop.run_until_complete(server.wait_closed())
    loop.stop()
    loop.close()
    exit(0)

client_shell.py

# implementation of a shell prompt (using Cmd module) to send message over TCP and process response
# standard imports
import socket
import cmd as cmd

class Shell(cmd.Cmd):
    def __init__(self, **kwargs):
        cmd.Cmd.__init__(self, **kwargs)

    def do_exit(self,*args):
        """
        Exits the shell gracefully
        :param args:
        :return:
        """
        print('Shutting down client...')
        return True

    def default(self, line):
        try:
            self._tcp_echo_client(line.encode())
        except SystemExit:
            pass

    def _tcp_echo_client(self, message):
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            print('Send: %r' % message)
            s.connect(('127.0.0.1', 8888))
            s.sendall(message)
            data = s.recv(1000)
            print('Received: %r' % data.decode())
            print('Close the socket')

if __name__ == '__main__':
    Shell().cmdloop()

Asyncio errors: CLI - python daemon example but has errors/undesired behavior on exit

shell.py

server.py

client.py

1 Answers1

server.py

client_shell.py