How to avoid blocking when reading from a socket with python asyncio?

Question

I'm trying to experiment a bit with python asyncio to improve in that area, for self teaching purposes I'm trying to connect to redis, send some commands and read the response, this can fall under generic "read a stream of data from some source". The problem I cannot solve is how to read data in chunks, since the connection is not being closed between server and client and the termination sequence \r\n could be met more than once. If I await when there is no more data of course the call will block until something else will be received.

class Client:
    def __init__(self, loop, host='127.0.0.1', port=6379):
        self.host = host
        self.port = port
        self.reader = None
        self.writer = None
        self.loop = loop

    @asyncio.coroutine
    def _connect(self):
        self.reader, self.writer = yield from asyncio.open_connection(
            self.host, self.port, loop=self.loop)

    async def read(self, b=4096):
        resp = b''
        while True:
            chunk = await self.reader.read(b)
            if chunk:
                resp += chunk
            else:
                break
        return resp

Let's pretend I want to read the response in chunks of 2 bytes (yes is stupid but it's just for this learning purpose) so:

loop = asyncio.get_event_loop()
client = Client(loop)
..... sends some commands here ....
resp = await client.read(2)

I cannot figure out how while not knowing the length of the server response the code can still be safe when the response is longer than the bytes read from the socket.

Related: [reader.readuntil](https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamReader.readuntil) — Vincent, Jun 19 '18 at 13:46
readuntil seems to do the job but still does not fit the case where one wants complete control over the size of the chunks — Mattia Procopio, Jun 19 '18 at 14:00
Change the termination sequence to something you'll only use when you terminate. — JakeJ, Jun 19 '18 at 14:33
How can the termination sequence be present more than once within the response? Isn't the definition of a _termination sequence_ something that terminates the response? Can you show how you would implement what you need with ordinary, blocking sockets? — user4815162342, Jun 19 '18 at 21:22
redis treats responses in a way where if you have more than one element, they are divided by `\r\n`, example `$6\r\nfoobar\r\n`. This is tho a more general problem, think if you're streaming a file for example — Mattia Procopio, Jun 20 '18 at 06:59
Normal sockets work the same way apparently, I was thinking that with some magic calling .recv(n) was returning None if there is nothing to stream but I was wrong, so this answer is related to normal sockets as well — Mattia Procopio, Jun 20 '18 at 07:07

score 0 · Accepted Answer · edited Jun 20 '18 at 11:17

0

I encountered a similar problem recently. My solution was to continue reading until a given character (or set of characters) is read. This is the same philosophy behind people saying "over" on walkie talkies when they are done talking. It is easier to just wait for the response to say that it is done talking.

While I haven't worked with the asyncio module before, I believe that the following code should solve your problem, assuming that the source of the input ends the response with whatever character (or string of characters) is indicated in the variable end_signal.

class Client:
    def __init__(self, loop, host='127.0.0.1', port=6379):
        self.host = host
        self.port = port
        self.reader = None
        self.writer = None
        self.loop = loop

    @asyncio.coroutine
    def _connect(self):
        self.reader, self.writer = yield from asyncio.open_connection(
            self.host, self.port, loop=self.loop)

    async def read(self, b=4096, end_signal = "10101101110111110"):
        resp = b''
        while True:
            chunk = await self.reader.read(b)
            resp += chunk
            if resp[-1*len(end_signal):] == end_signal:
                break
        return resp

edited Jun 20 '18 at 11:17

Mattia Procopio

631
8
16

answered Jun 19 '18 at 14:32

S. Mansfield

90
5

Code? This isn't really an answer yet, although it has the potential to possibly be one. – Jared Smith Jun 19 '18 at 14:33
The problem and hence my question is how to avoid blocking if the termination sequence is not known or like in redis it can be present multiple time within the response – Mattia Procopio Jun 19 '18 at 18:05
Isn't this code a reimplementation of [`StreamReader.readuntil`](https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamReader.readuntil)? – user4815162342 Jun 19 '18 at 21:23
No, that's the right way of doing it. readuntil will read until sequence is found, that means if the response is `$6\r\nabcdef\r\n` it will truncate the response while checking if the chunk ends with what you expect will give the desired result – Mattia Procopio Jun 20 '18 at 18:46

How to avoid blocking when reading from a socket with python asyncio?

1 Answers1