How to append unique incoming messages to a list?

Question

I am using a RFID reader to receive messages. I'm trying to avoid duplicated message by appending them to emptylist = [] and don't further append them if they exist in the list. Below are my code:

import paho.mqtt.client as mqtt

def on_connect(client, userdata, flags, rc):
    print("Connected with result code " + str(rc))
    client.subscribe("/whatver/#")

def on_message(client, userdata, msg): # when PUBLISH msg is rcvd frm the server
    payloadjson = json.loads(msg.payload.decode('utf-8'))
    line = payloadjson["value"].split(',')
    epc =  line[1]


    payload = {'a': epc[11:35], 'b':payloadjson["devicename"], 'c':payloadjson["sensorname"]}

    emptylist = []
    emptylist.append(payload)

    if payload not in emptylist:
        emptylist.append(payload)

    print (emptylist)

test = mqtt.Client(protocol = mqtt.MQTTv31)
test.connect(host=_host, port=1883, keepalive=60, bind_address="")
test.on_connect = on_connect
test.on_message = on_message
test.loop_forever()

However I am getting output as below, which shows that payload is appended to the emptylist, but is stored into multiple separated lists. And it keeps printing the same output if the RFID reader is still reading the same tag. I would like to remove the duplicated messages and retain only one even if the RFID reader reads the same tag.

[{'b': 'READERBASIN.3', 'c': 'ANTENNA.2', 'a': '000000000000130000624462'}] # from Tag A
[{'b': 'READERBASIN.3', 'c': 'ANTENNA.2', 'a': 'abcxxx000000130000627000'}] # from Tag B
[{'b': 'READERBASIN.3', 'c': 'ANTENNA.2', 'a': '000000000000130000624462'}] # from Tag A
[{'b': 'READERBASIN.3', 'c': 'ANTENNA.2', 'a': '000000000000130000624462'}] # from Tag A
.
.
.

How can I fix this?

score 2 · Accepted Answer · answered Mar 28 '16 at 07:27

As the other answers mention, a set is a great way to avoid storing or printing duplicates. You can do this (less efficiently) using lists, however, the logic in your current code is incorrect. It creates a fresh empty emptylist every time that on_message is called, and then appends the new payload to it, so the following if payload not in emptylist: doesn't make much sense.

You need to initialize emptylist outside of the on_message callback. Eg,

emptylist = []

def on_message(client, userdata, msg): # when PUBLISH msg is rcvd frm the server
    payloadjson = json.loads(msg.payload.decode('utf-8'))
    line = payloadjson["value"].split(',')
    epc =  line[1]

    payload = {
        'a': epc[11:35], 
        'b':payloadjson["devicename"], 
        'c':payloadjson["sensorname"],
    }

    if payload not in emptylist:
        emptylist.append(payload)
        print (emptylist)

That will only append & print the new payload if it's not already in emptylist.

To do the same thing using a set:

emptylist = set()

def on_message(client, userdata, msg): # when PUBLISH msg is rcvd frm the server
    payloadjson = json.loads(msg.payload.decode('utf-8'))
    line = payloadjson["value"].split(',')
    epc =  line[1]

    payload = {
        'a': epc[11:35], 
        'b':payloadjson["devicename"], 
        'c':payloadjson["sensorname"],
    }

    if payload not in emptylist:
        emptylist.add(payload)
        print (emptylist)

FWIW, emptylist is a slightly confusing variable name. :)

Thanks for this great tip! – jacky_learns_to_code Mar 28 '16 at 07:38 — jacky_learns_to_code, Mar 28 '16 at 07:38

krishv · Answer 2 · 2016-03-28T09:08:53.847

0

Either you can check for existence while appending as below:

if payload not in emptylist:
   emptylist.append(payload)

or

You can choose set(emptylist) on your final output.

edited Mar 28 '16 at 09:08

answered Mar 28 '16 at 06:45

krishv

36
4

How to append unique incoming messages to a list?

2 Answers2