3

Is there an elegant way to pull one message off the broker without having to:

  1. subscribe
  2. create an on_message()
  3. receive the message
  4. unsubscribe

I ask because we are using a json message which has multiple fields. When new data comes in I want to ONLY update that particular field in the json message but not remove the rest of the data. Since we have a TON of these json topics, we don't really want to keep all of them in program memory (also in case the program has to be relaunched). On top of that, this program could be running for months without supervision.

So ideally, I'd like to post the json message to an ID'd topic with the retain flag set to True. Then when new data comes in for that ID, I do a pull of the info on that topic, update that particular field in the json message and repost to the same topic.

I can post example code but I'm hoping there is a simple function that I am unaware of.

Thanks, in advance, for any suggestions.

hardillb
  • 54,545
  • 11
  • 67
  • 105
Scott
  • 61
  • 2
  • 9

2 Answers2

7

The Paho Python client comes with a set of help classes that do this single shot type of pattern for you.

Doc here

e.g. the following connects to a broker, subscribes to a topic and returns on receipt of the first message on that topic.

import paho.mqtt.subscribe as subscribe

msg = subscribe.simple("paho/test/simple", hostname="mqtt.eclipse.org")
print("%s %s" % (msg.topic, msg.payload))

And the matching publish call:

import paho.mqtt.publish as publish

publish.single("paho/test/single", "payload", hostname="mqtt.eclipse.org")
hardillb
  • 54,545
  • 11
  • 67
  • 105
  • That might be EXACTLY what I am looking for ... I knew of the single publish but not the subscribe ... I'll test it out and see what happens. Thanks. – Scott May 01 '20 at 19:23
  • I am not sure if that is what you need. This is basically only the "normal" MQTT functionality wrapped in a simple function. – Joe May 01 '20 at 20:18
0

I don't think that is possible. You say "When new data comes in..." That's exacty why you need to subscribe and use the callback function. That's basically a "pull when something is actually there".

Just to get an idea of how it should work: you are sending that json message via MQTT, right? And you are re-sending it when it changes?

But you don't have to keep them all in the RAM. You could use a retained message in combination with a fixed topic (not ID'ed) and send the ID in the message.

If you use retained messages with ID'ed topics, that might fill the memory.

What does the ID stand for? A uniqie number? Something like a timestamp? A hash? The sender?

I think you can solve that problem by clearly separating your things, e.g. say in data and message, where data is something you maintain in Python (e.g. a database or something in RAM) and message is something that you acually send / receive via MQTT.

Then you can add / send / update data depending on what is received in MQTT and you don't have to send / update the complete set.

Joe
  • 6,758
  • 2
  • 26
  • 47
  • My mistake ... I didn't explain it correctly. I have TCP connections which receives data about unique_id modules. These modules have a bunch of different parameters which we get via a proprietary string. We parse those strings ... and update the new info for them on the broker. What's worse is these modules are continually being created and destroyed. So, we don't have a finite number of these modules (each module has it's own topic). So the application memory usage could grow very quickly. – Scott May 01 '20 at 19:08
  • Pseudo code: tcp_socket_messge(msg): id, new_params = parse_socket_msg(msg) write_to_broker(id, new_params) write_to_broker(id, new): topic = "module/" out = json.dump(new) client.publish(topic, out, retain=True) Say (hypothetically) module ID#: 001, has 35 parameters and one of them is "temperature" ... then when I get the temperature data I will end up overwriting the entire dataset of 24 parameters when updating only one for that ID. Note, the number (and names) of parameters are dynamic for each module ... – Scott May 01 '20 at 19:21
  • sorry ... I didn't know how to add code to comments – Scott May 01 '20 at 19:30
  • Added some suggestions to the question. – Joe May 02 '20 at 04:37