2

I will use java mail api to handle mails like thunderbird etc. I have to fetch mails having 1000 messages. My design will be: When user performs a synch on a folder, i will get all uids of the messages in the folder:

Message[] msgs = ufolder.getMessagesByUID(1, UIDFolder.LASTUID);
// Use a suitable FetchProfile
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add(FetchProfile.Item.FLAGS);

I wil then compare the list of uids with the list stored in my db. For the deleted ones, for example a message is not in the folder but in the db, i will mark it as deleted. For the new ones, for example a message is in the folder but not in the db, i will mark as possible new. But, because messageuids are not safe (can be changed by the mail server on some cases), for the new mails, i will use additioanlly a custom hash value build from message id in the header + subject + receivedate and build a md5 hash. Only for the possible new mails i will use this hash and catch new mails. For the moved messages, because their uids will be changed in the new folder, it will be flagged as deleted in the first and will be a new message in new folder, but the message will have same custom hash value becaue message id in the header and other properties will remain same duing the movement.

Question about performance issue: On each click on folder (folder synch) i will do the compare operation of all uids in the folder with the local uid list stored in the db to learn the deleted ones. I could not find another better way to accomplish this. As you know, thunderbird catches immediately a deleted message without relogin, even if the folder is large and the deleted message is very old (5 years). I think thunderbird also compares all message uids in that folder with a list stored locally.

How can i implement a better mechanism for the synch for a better performance? Does thunderbird apply a different approach? How can thunderbird accomplish it so quickly?

If we were interested only for the new messages, i could have kept last stored uid and only compare the new messages later than that, but for the deleted ones, i already have to compare full folder. Additionally, UIDNEXT value is always -1 in my mail server, if it were set correctly, it will not help to get deleted ones again, a full compare is a must i think, am I wrong?

Note: I canot not use or add message listeners because the appliaction is server-client based and the mail handling task is on the server side and we do not support threads listeners etc. The events should be triggered from the client and the request is being processed on ther server and a response is returned and client handles the response on the gui.

benchpresser
  • 2,171
  • 2
  • 23
  • 41

2 Answers2

3

What you want is called condstore or quick resync, RFC7162 in both cases. That's what Thunderbird uses.

That's a pair of extensions support commands like "give me all the UIDs that have changed since last time I connected", "tell me what's been deleted" and so on.

arnt
  • 8,949
  • 5
  • 24
  • 32
  • 1
    Yes but it says it is not widely supported and whole structure should not be built on it: http://stackoverflow.com/questions/23176247/is-imap-condstore-widely-supported I think i should keep last stored uid and compare only that for the new mails. Or should I use getNextUID()? But on mail server it is always -1. On the other hand, suppose i store the 10.mail has the uid 100 and i keep this value. On the nexy fetch i say fetch mails starting with UID 100 and newer. But the mail with uid has been deleted before I fetch? Does imap support get uids larger than x (are uids awlays incremental?) – benchpresser May 11 '15 at 09:10
  • 2
    IMAP UIDs always increase so long as the mailbox' uidvalidity value remains the same. And it's quite right, it's not widely supported. But if you want to do the thing you want to do, it's the most widely supported way (and in practice the only way). Various servers don't support it, such as mail.ru, which effectively means that if Russian end-users are a significant part of your target market, then the feature you want is not practically implementable. – arnt May 11 '15 at 09:46
0

If you can't use threads to listen for these events from the mail server, your options are very limited. Probably the best thing you can do is limit the resynchronization to the messages that are visible to the client.

Bill Shannon
  • 29,579
  • 6
  • 38
  • 40
  • Do mail clients for example tbird register listeners and getnotified upon deletion of mails or do they perform a full check of uids and compare with a list of uids stored locally of current folder when user triggers refresh or synch? – benchpresser May 10 '15 at 20:43
  • Well, they don't use JavaMail, so who knows how they really handle this, but I believe they watch for the same events that are notified using listeners in JavaMail. – Bill Shannon May 11 '15 at 03:24
  • 1
    But, on initial phase, when first login, they have to get somehow a list and compare with local. On the other hand, we can not rely to listeners always, and imap-idle supports offline working. And what happens, we are connected to mail server and listening, a mail is received but our network is disconnected for short period, mail server notified us but we did not get it. What happens when we reconnect? Does the mailserver send the event again? Does it wait ack from us and if not received does it send the event again? – benchpresser May 11 '15 at 07:01
  • 1
    It's complicated. To see what Thunderbird does, enable [session logging](http://kb.mozillazine.org/Session_logging_for_mail/news). You'll need to read the [IMAP RFC](https://tools.ietf.org/html/rfc3501) carefully. You'll also find [RFC 1733](https://tools.ietf.org/html/rfc1733) and [RFC 4549](https://tools.ietf.org/html/rfc4549) helpful. There's also lots of IMAP extensions (e.g., CONDSTORE, QRESYNC) that will help, but support of the extensions by servers is inconsistent. – Bill Shannon May 11 '15 at 18:35