0

I am implementing a MQTT worker in python with paho-mqtt.

Are all the on_message() multi threaded in different threads, so that if one of the task is time consuming, other messages can still be processed?

If not, how to achieve this behaviour?

Alexander Farber
  • 21,519
  • 75
  • 241
  • 416
user3684457
  • 39
  • 1
  • 7

1 Answers1

4

The python client doesn't actually start any threads, that's why you have to call the loop function to handle network events.

In Java you would use the onMessage callback to put the incoming message on to a local queue that a separate pool of threads will handle.

Python doesn't have native threading support but does have support for spawning processes to act like threads. Details of the multiprocessing can be found here:

https://docs.python.org/2.7/library/multiprocessing.html

EDIT: On looking closer at the paho python code a little closer it appears it can actually start a new thread (using the loop_start() function) to handle the network side of things previously requiring the loop functions. This does not change the fact the all calls to the on_message callback will happen on this thread. If you need to do large amounts of work in this callback you should definitely look spinning up a pool of new threads to do this work.

http://www.tutorialspoint.com/python/python_multithreading.htm

hardillb
  • 54,545
  • 11
  • 67
  • 105
  • do you recommend me threading or async (like twisted) if I need to do heavy processing on the message? – user3684457 Dec 19 '14 at 23:36
  • If processing a message could take l take than the gap between messages then you should be looking at multiple threads. But remember this can only happen in bursts else you will just end up with messages either waiting on a local queue or a ever increasing number of threads – hardillb Dec 20 '14 at 07:56
  • Is there a way to improve how python manages threads? Is there a limit I should consider (no more that x threads at the same time?)? – user3684457 Dec 22 '14 at 09:17