1

I am trying to create a Python producer and consumer using 2 asynchronous functions. 1 of the functions is a callback getting data from server (producer), the other is a timed callback that executes every N seconds (consumer). I am trying to use ZMQ/Reactor for this as I would like to avoid overhead of a timer thread and just use the event loop. Does the ZMQ/Reactor event loop execute each event in order they arrive or is there a chance that the consumer function could be context switched to the producer function at any time?

Simplified Example:
The main issue is that I cannot pop all items from a list and clear it atomically

queue = []
def producer():
   # gets data from server
   queue.append(single_record)

def consumer():
   records = list(queue)
   # inserts records into db
   queue.clear()
Steve Freed
  • 91
  • 1
  • 11

1 Answers1

0

Q : "Can ZMQ context switch during an event?"

Well,
let's start from the fathers of the Zen of Zero - you were never promised by Pieter HINTJENS' & Martin SUSTRIK's architecture evangelisation any such warranty of receiving a message at all, the less a warranty to get all of them the less "in order".

There is one and only one warranty in the Zen-of-Zero : if we indeed do .recv a message, it is sure to be a binary-identical replica of the original one, that was dispatched to be sent over ZeroMQ.

The fact that the ZeroMQ Context()-instance is an independent, asynchronous MQ-engine, does not help in this and knowing that Python switches its internal GIL-lock ownership each hundred of pseudo-instructions in Py2.x or each 5[ms] in Py3.+ does not help your expressed need to get any such warranty either, does it?

You need some other add-on mechanics, that glue-in the gaps of Python and that bridge the missing warranties over the territories of Zen-of-Zero.

halfer
  • 19,824
  • 17
  • 99
  • 186
user3666197
  • 1
  • 6
  • 50
  • 92
  • 1
    I thought you schedule callbacks on the event loop and they execute in order? I am also using Twisted Reactor with ZMQ – Steve Freed Sep 10 '20 at 02:41