How to check duplicates are not being added to a LinkedBlockingQueue?

Question

I am receiving messages from multiple gmail accounts using the Java mail API. The different accounts are being processed by different threads, and I am using a LinkedBlockingQueue to store the emails. However I do not want the same email repeatedly being added to the Queue. This is the code I have so far:

public synchronized void readMail(){
    try {
        boolean alreadyAdded = false;
        Folder inbox = store.getFolder("Inbox");
        inbox.open(Folder.READ_ONLY);
        Message [] received = inbox.getMessages();

        if(messages.isEmpty()){
            for(Message newMessage:received){
                System.out.println("Queue empty, adding messages");
                messages.put(newMessage);
            }
        }

        else{
            for(Message existingMessage:messages){
                for(Message newMessage:received){
                    if (alreadyAdded == true)
                        break;

                    else{
                        if(existingMessage.getSubject().equals(newMessage.getSubject())){
                            alreadyAdded = true;
                            System.out.println("boolean changed to true, message "+newMessage.getSubject()+"won't be added");
                        }

                        else{
                            alreadyAdded = false;
                            System.out.println("Non-duplicate message "+newMessage.getSubject());
                            messages.put(newMessage);
                        }
                    }
                }
            }
        }
    } 
    catch (MessagingException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

The problem I am having is in the else block after the if checking if the Queue is empty. I want it to check through the set of messages that has just been read in and compare those to the messages already in the Queue. If the message is in the Queue don't add it again. I cannot simply use .contains(), as each time messages are downloaded, they are given a different memory location, so although the Message object may in fact be the same (e.g. has the same subject, content, etc) it will not have the same signature (e.g the first time it is downloaded it may be Messagehgshsh676767 but the next time it could be Messageyyetwt8965).

I have hit a brick wall, can anyone suggest a way to make sure no duplicates are added?

What if you have a duplicate Message which is being processed, and not in the queue? I suggest you are better off keeping track of the message performed and remove duplicates by comparing to tasks performed, not those currently in the queue. — Peter Lawrey, Nov 13 '12 at 17:05
The key to all of this is defining exactly what you mean by "same message". In general, two messages with the same subject are NOT the same message. Once you figure out what "same message" means to you, any of the suggestions below will work. — Bill Shannon, Nov 13 '12 at 19:19

ShyJ · Answer 1 · 2012-11-13T17:25:00.050

You can create a wrapper class for your messages, which will contain a proper equals method, based on e.g. subject

public class MyMessage {
    final private Message msg;

    public MyMessage (final Message msg) {
        this.msg = msg;
    }

    public boolean equals (final Object other) {
        if (!(other instanceof MyMessage)) {
            return false;
        }

        final MyMessage otherMessage = (MyMessage) other;
        return msg.getSubject ().equals (otherMessage.getSubject ());
    }

    public Message getMessage () {
        return msg;
    }
}

If you do not care about the order you can use some thread-safe Set implementation

final Set<MyMessage> messages = Collections.synchronizedSet (new HashSet<MyMessage> ());

Message [] received = inbox.getMessages();
for (final Message msg : reveived) {
    messages.add (msg);
}

This way you won't have duplicates.

If you care about the order use some SortedSet , e.g.:

public class MyMessage implements Comparable<MyMessage> {
    ... //the same as above

    public int compareTo (final MyMessage otherMessage) {
        return msg.getReceivedDate ().compareTo (otherMessage.getReceivedDate ());
    }
}


final Set<MyMessage> messages = Collections.synchronizedSet (new TreeSet<MyMessage> ());

+1 One note, is that if the OP wanted to use a thread-safe collection like LBQ, and have it be naturally ordered, then I would use `PriorityBlockingQueue` — John Vint, Nov 13 '12 at 18:42

Random42 · Answer 2 · 2012-11-13T17:17:30.770

-1

if(!queue.contains(element)) {
    queue.add(element);
}

If you want a Set you might extend the LinkedBlockingQueue and override add:

public boolean add(E e)
    if(!this.contains(e)) {
        return super.add(e);
    } else {
        return false;
    }
}

but you should either make it and use it localy or override it properly - all the methods and constructors that can add elements to it.

Regarding the contains issue, create a wrapper over the received Message and properly implement the public boolean equals(Object o) method. When you receive the message put it in this wrapper and put this wrapper in the collection.

edited Nov 13 '12 at 17:17

answered Nov 13 '12 at 17:10

Random42

8,989
6
55
86

There would be a race condition in such statement. – ShyJ Nov 13 '12 at 17:11
Like I said, I cannot use .contains due to the fact the Message objects of messages that are the same, will not be the same – user1821475 Nov 13 '12 at 17:14
@user1821475 I'm sorry I did not notice; you can create a wrapper for it and implement the equals method there. – Random42 Nov 13 '12 at 17:18
@m3th0dman Since he's using `BlockingQueue` I assumed there is some background thread reading off that queue. So the synchronization on the `readMail` method is not enough. – ShyJ Nov 13 '12 at 21:38

How to check duplicates are not being added to a LinkedBlockingQueue?

2 Answers2