how to store and parse real-time streaming data in mqtt paho for every Nth minutes (python)

Question

Im working on a real-time data streaming project to parse and store the data every Nth minute. My objective is to throw away the very first minute of data(as buffer) and store every 4 minutes long of data from the server. The data will then be parsed to other functions to be clustered and calculation(functions not included here).

I have initialized the condition at 'on_message' function and data parse it inside that fuction. I dont think my structuring and calling is the correct way to achieve my objective. Please let me know if you require any additional details.

on_message

def on_message(r_c_client, userdata, message):

    if (message.topic == "scanning"):

     c = datetime.now().time()
     current = (c.hour * 60 + c.minute) * 60 + c.second

     time.sleep(60) #initial delay

     data = json.loads(message.payload.decode("utf-8"))
     x = data['host']
     y = data['data']

     hostList = store(x, y)

     while (current>=total_Time ):
      #time.sleep(60) #initial delay


        nodeList = listToDf(hostList)


        nodeDf= df_reformat(nodeList)
        print clustering_results_reformat(process_startTime, nodeDf)

store function

def store(host, data):





  if host in hostList:
      hostList[host].append(data)

  else:
      hostList[host] = [data]

  return hostList

main

global process_startTime

t = datetime.now().time()

process_startTime = (t.hour * 60 + t.minute) * 60 + t.second

total_Time = process_startTime + 300 #4 minutes + 1 minute 

print t , process_startTime

broker_address = '10.10.0.100'
c_client = mqtt.Client("trilateration")
c_client.on_connect = on_connect


c_client.on_message = on_message
c_client.on_subscribe = on_subscribe


c_client.connect(broker_address, 1883)

c_client.loop_forever()

Answer 1

First up, you should NEVER block (sleep) in the on_message function, this function is called for EVERY message that is received, if you sleep then the system will have to wait that length of time before moving on to the next message.

Next, you need to keep track of the start time outside the on_message function, you can then compare the current time to this value for each message and decide if you are going to keep/process it or not.

def on_message(r_c_client, userdata, message):
  global process_startTime

  if (message.topic == "scanning"):
   c = datetime.now().time()
   current = (c.hour * 60 + c.minute) * 60 + c.second

   if (current<=total_Time and current>=(process_startTime + 60)):
     data = json.loads(message.payload.decode("utf-8"))
     x = data['host']
     y = data['data']

     hostList = store(x, y)

The main should look something like this:

global process_startTime

t = datetime.now().time()

process_startTime = (t.hour * 60 + t.minute) * 60 + t.second
total_Time = process_startTime + 300 #4 minutes + 1 minute 
print t , process_startTime

broker_address = '10.10.0.100'
c_client = mqtt.Client("trilateration")
c_client.on_connect = on_connect

c_client.on_message = on_message
c_client.on_subscribe = on_subscribe
c_client.connect(broker_address, 1883)

while (True):
  c_client.loop()
  c = datetime.now().time()
  current = (c.hour * 60 + c.minute) * 60 + c.second
  if (current >= total_Time):
    nodeList = listToDf(hostList)
    nodeDf= df_reformat(nodeList)
    print clustering_results_reformat(process_startTime, nodeDf)
  time.sleep(1)