How can Postfix filter email (DKIM) without keeping the message in memory and without writing it to disc twice?

Question

I need to DKIM sign possibly huge emails (up to 150MB). I’m running Postfix and so far want to keep that MTA.

Conceptually DKIM needs to go over the email twice: once to calculate and sign the checksum and once to write it out with the result of the previous step in the headers.¹

A DKIM signer can do this by either keeping the message in memory (a no-go for me) or write it to a file.

For the task at hand I want to use a Postfix (filter) mechanism that allows me to do that without keeping the message in memory and without having it written to disc twice!

So far I see that the after-queue content filter mechanism forces you to write the email to disc again, and for no good reason! It should instead pass a seekable file descriptor to filter’s stdin, but the implementation does not do it.

The alternative, the before-queue milter, is insufficiently documented for me to see if it avoids keeping the message in memory and avoids writing the original mail to file twice. – This is why I have opendkim in my tags: maybe those experts know how the milter API can avoid and how opendkim indeed does avoid these pitfalls.

---

_{¹...as Posix file systems have no prepend operation}

Answer 1

Postfix queue files are not flat mails. Adding a header does not require a rewrite. To take advantage of that use the milter interface. The answers I got from postfix-users make me believe mail is not kept in memory during milter processing either. At least not by Postfix.

Using the pipe mechanism with the after-queue content filter would not do it as mentioned in the question. A write out to file to avoid the mail in memory would probably be reasonable enough though and better than keeping it in memory.

While the milter interface is good enough for DKIM, I’ld like to list it’s shortcomings (all of them could have been avoided):

• you can not modify neither header nor body(-parts) before the entire message is received
  • no proper in transit piping
• you can not back reference no header and no body once you are allowed to replace/modify content
  • milter client needs to keep a copy during reception phase if it needs the information
• the body can only be replaced in its entirety
• header substitution/deletion require name and index, but milter server is not passing the index number (or any other opaque unique reference)
  • milter client needs to count headers for any header it might later decide to replace

Postfix has some shortcomings as well:

• postfix offers 3 filter mechanisms at 2 positions
  • you can not mix and match mechanism and position
  • the most appropriate mechanism for DKIM is milter
  • the most appropriate place for DKIM signing is after queue
  • after queue milter is not available
    • within limits that would be possible
    • postfix can actually already fake SMTP/milter environments to make milters work in new areas (“non-smtp-milter”)
• no mechanism exploits all benefits of what would be possible with the current queue data structure
  • not needed for DKIM, though; just saying