5

I have a zmq server that subscribes to several publisher (but in this example only to one)

After making the server async by creating a coroutine and receiving a serialized multipart message deserialization with pickle.loads and json.loads both seem not to work. (I do not even get an error message)

When I make my application without the async coroutine serialization works like a charm. If I omit the deserialization the server also works perfectly.

So I am wondering if for some reason json and pickle api need to be adopted for asyncio and if so: how is this usually being done? 

async def receiver():
    print("1")
    socket =  ctx.socket(zmq.SUB)
    with ctx.socket(zmq.SUB) as socket:
        print("2")
        socket.setsockopt(zmq.SUBSCRIBE, b"")
        print("3")
        for url in urls:
            socket.connect(url)
        print("4")
        poller = Poller()
        poller.register(socket, zmq.POLLIN)
        while True:
            events = await poller.poll()
            if socket in dict(events):
                print("5")
                items = await socket.recv_multipart()
                res = b"".join(items)
                s = res.decode(encoding="utf-8")
                print(s)
                t = json.loads(s)
                print(t)
                print("6")

if I omit the lines t = json.loads(s)and print(t) I will get the output 6 and when receiving more messages I will get the output 5 and 6 for every received message. If I keep those lines I don't get the output 6 and don't receive any more messages

the publisher code basically looks like that: 

ctx = zmq.Context()
socket = ctx.socket(zmq.PUB)
socket.bind(addr)

f = open("foo.json", "r")
for line in f:
    jsn = json.loads(line)
    for el in jsn["somekey"]["someOtherKey"]:
        data = {"data":el,"some more metadata":"here is the meta data"}
        b = json.dumps(data).encode()
        #b = pickle.dumps(data,protocol=pickle.HIGHEST_PROTOCOL)
        socket.send_multipart(b)

I should state that the publisher runs on python2 and the receiver on python3. The publisher code won't run on python3 since I receive the following stack trace with python3 (which also does not make sense to me)

 Traceback (most recent call last):
 File ".../python3.6/site-packages/zmq/sugar/socket.py", line 356, in send_multipart
    _buffer_type(msg)
 TypeError: memoryview: a bytes-like object is required, not 'int'

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
 File "Publisher2.py", line 39, in <module>
    main("tcp://127.0.0.1:5556", "mockPublisher")
 File "Publisher2.py", line 28, in main
    socket.send_multipart(b)
 File ".../python3.6/site-packages/zmq/sugar/socket.py", line 363, in send_multipart
    i, rmsg,
 TypeError: Frame 0 (123) does not support the buffer interface.
user4815162342
  • 141,790
  • 18
  • 296
  • 355
Rene Pickhardt
  • 692
  • 5
  • 17
  • 2
    The `loads` most likely raises an exception which you don't see because the coroutine runs in the background and no one awaits it. Add `try: ... except: import traceback; traceback.print_exc()` arround the offending call and you'll have more information. – user4815162342 Aug 01 '18 at 08:57
  • The publisher fails on Python 3 because `send_multipart` accepts a _sequence_ of buffers to send as a multipart message. You're sending it a single buffer, which it tries to treat as a sequence and fails because it finds individual bytes inside. The same code only accidentally works on Python 2, because in Python 2 iterating over a (byte) string produces single-element (byte) strings, so you're sending a corret but highly inefficient multi-part message with each part consisting of a single byte. I guess you need to use `socket.send(b)` or at least `socket.send_multipart([b])`. – user4815162342 Aug 01 '18 at 09:02
  • I think I was actually looking for the `try: ... except:` to resolve the problem of not seeing tracebacks when using coroutines. Since I reorganized my code I missed importing json / pickle which is obviously why loads silently failed every time. Thanks! – Rene Pickhardt Aug 01 '18 at 11:02

0 Answers0