0

I am using the maildir (maildir++, actually) email backend format for storing messages on a custom IMAP server. I have a good understanding of the relevant email protocols and maildir format, but this is one question I've had for a while that's been bothering me a bit. The maildir specification, relatively unchanged for decades now, basically says:

When a cognizant maildir-reading process (either a POP or IMAP server, or a mail user agent acting locally) finds messages in the new directory, it must move them to cur. It is just a means to notify the user "you have X new messages".[11] This moving needs to be done using rename(), as the non-atomic link-then-unlink technique may result in duplicated messages. An informational suffix is appended to filenames at this stage. It consists of a colon (to separate the unique part of the filename from the actual information), a "2", a comma and various flags. The "2" specifies the version of the information that follows the comma. "2" is the only currently officially specified version, "1" being an experimental version. The specification defines flags that show whether the message has been read, deleted and so on: the initial (capital) letter of "Passed", "Replied", "Seen", "Trashed", "Draft", and "Flagged".[7] Dovecot uses lowercase letters to match 26 IMAP keywords,[6] which may include standardised keywords, such as $MDNSent, and user-defined flags. - https://en.wikipedia.org/wiki/Maildir#Technical_operation

When you move a file from new to cur, you have to change its name from uniq to uniq:info - http://cr.yp.to/proto/maildir.html

For the most part, this seems to work fairly well. Any mail delivering process can just dump messages in new, and the IMAP server can find these in new and move them to cur. Overall, it seems very coherent; I do not have to store the \Recent flag explicitly. It is set implicitly if the message is in new but not in cur.

However, there are a few problems I've identified over time with the maildir format:

  • UIDs are not assigned until a message is moved from new to cur. In other words, messages with the \Recent flag do not have a UID. This is generally not an issue, but it does make it impossible to provide UIDs for the COPYUID response, and appended messages must be moved to cur immediately to provide the APPENDUID, causing them to lose their \Recent status.

  • Flags cannot be stored for \Recent messages. Most IMAP servers, however, don't seem to have an issue with this. Appended emails (e.g. to Sent) can still be marked as seen and recent at the same time. In the maildir format, however, these would seem to be mutually exclusive: a message cannot be recent and have other flags at the same time.

  • Sieve allows users to add flags to messages. But it may make sense to keep the message as "Recent", but maildir does not facilitate this.

This is all at least based on my interpretation of the base maildir specs. I'm wondering if maildir has evolved in some way to be able to handle these? The most obvious thing to me would be to simply do all the "tagging" when a message is put into new, not cur. e.g. assign the UID then and allow it to have flags then. However, this would seemingly break mail delivering applications using the maildir format that just expect to be able to dump messages there. At the very least, it seems inconsistent, but I don't know how much of a problem would be.

Is there a recommended approach of supporting flags and UIDs for \Recent messages with maildir backends? Everything I can find on maildir spec-wise is about 20 years old and doesn't really detail how to deal with these edge cases. I'm not looking for code, just a general algorithm like the original maildir spec prescribes.

InterLinked
  • 1,247
  • 2
  • 18
  • 50

1 Answers1

0

The evolution that's happened is that clients mostly disregard \recent and various servers don't set it, ever. \recent provides a poor experience when using many modern clients, since messages lose that flag when a phone leaves a WLAN and reconnects via mobile data, which happens all the time. That looks unprovoked to people, puzzling, so clients mostly just don't show \recent.

A maildir-based server that ignores \recent can move messages to cur as soon as it sees them, assign an UID then, set flags, etc.

arnt
  • 8,949
  • 5
  • 24
  • 32
  • I know it's kind of flaky, but most commercial providers seem to have proper support for `\Recent` - the only notable exception is Gmail. Thus, some servers do support it properly. Maybe it's possible they're all not using maildir, I'm just wondering if there is an established way to provide `\Recent` with a maildir backend. I don't just want to make up my own hack here if I can avoid it. In my case, I do use `\Recent` so I don't want to ignore it. – InterLinked Jul 28 '23 at 14:24
  • Yes. AFAICT, before the smartphone age most server authors wrote the necessary code. They still have the code. – arnt Jul 31 '23 at 08:21
  • Thanks - and you have seen this with maildir backends specifically before? Is there any type of standard approach you've seen? – InterLinked Jul 31 '23 at 12:02
  • 1
    It was discussed on one of the mailing lists 20-25 years ago, possibly on the old imap-protocol list. IIRC one of the servers used an approach where an IMAP server process would move new messages to cur as soon as it saw them and make a really long file name that encoded the UID and lots more. The process that did the moving would remember what it had moved and serve `\recent` based on RAM state. Don't remember anything about the file name encoding and I... think the server in question wasn't courier-imap. It's long ago. – arnt Jul 31 '23 at 12:33
  • Thanks, do you know off hand where I could find public archives of that list? I found http://mailman12.u.washington.edu/pipermail/imap-protocol/ but that seems to be only the past few years, and the link on https://www.imapwiki.org/ImapProtocolList is a 404 – InterLinked Jul 31 '23 at 13:44