Compress old emails with dovecot

Question

How can I configure dovecot to compress/archive old emails (say 6 months old) with the users still being able to read them, to save some precious space on the server?

Answer 1

The configuration to compress emails is a two step process:

1. We need to configure dovecot to be able to read compressed emails
2. We need to setup a cronjob to compress emails

Dovecot configuration

Create a configuration file for dovecot to enable the zlib plugin

Dovecot configuration files are usually found in /etc/dovecot/conf.d/

You can create a file named 35-zlib.conf in this directory with this content

# Enable Zlib for imap
protocol imap {
  mail_plugins = $mail_plugins zlib
}
# Enable Zlib for pop3
protocol pop3 {
  mail_plugins = $mail_plugins zlib
}
# Increase memory allowed for imap as it costs more to read compressed files
service imap {
  vsz_limit = 1024MB
}

Make sure your configuration is correct with doveconf -n, then restart dovecot using service dovecot restart

Daily cronjob

Next we need to run a script every day to compress emails that are older than 6 months

Hopefully this can be done with a [verbose] find command

In /etc/cron.daily/ create a file, let's call it archive-email

Write this script in it

#!/bin/bash
find /var/qmail/mailnames -mtime +182 -type f -regex ".*\.$HOSTNAME,.*S=.*,[a-yA-Y]*" ! -name "*\.gz*" -print0 |
 while read -rd $'\0' FULLPATH
  do
    echo "${FULLPATH}"
    if file "${FULLPATH}" | grep compressed -q; then
      mv "${FULLPATH}" "${FULLPATH}Z" # Already compressed
    else
      BASENAME=$(basename "${FULLPATH}")
      cp -a "${FULLPATH}" "/tmp/${BASENAME}" # Move to tmp
      gzip -f "/tmp/${BASENAME}" # Compress
      if [[ -f ${FULLPATH} ]]; then # Make sure the original file has not been deleted while we were compressing
        mv "/tmp/${BASENAME}.gz" "${FULLPATH}Z" && rm "${FULLPATH}"
      else
        rm "/tmp/${BASENAME}.gz"
        continue
      fi
    fi
    echo "${FULLPATH}" >> /var/log/archived-emails.log
  done

Don't forget to run chmod +x on the created file to make it executable

/var/qmail/mailnames is usually the root folder in which emails are stored, it may depend on your configuration

The -mtime +182 modifier allows us to find only files created more than 182 days ago (basically 6 months), modify it accordingly

With the-type f -regex ".*\.$HOSTNAME,.*S=.*,[a-yA-Y]*" modifier, we want to find only email files that don't have the Z flag in them, which are usually named in this format 1505740244.M351559P24632.my.domain.com,S=38204,W=40910:2,S More information on the maildir format here

Finally we copy the emails to compress to the \tmp dir to avoid conflict while compressing, we run the gzip command on the file and move them again if the file still exists in their original location adding a Z flag in their name to indicate they are compressed so we don't gzip them again and finally remove the original file

This process has been implemented according to the guidelines of the dovecot wiki but the maildirlock utility has a bug since v2 that has never been fixed, so I'm not using it, however it's unlikely it will cause issues

We also log all the successfully archived files in /var/log/archived-emails.log

You can run the command in your terminal first without exec to make sure you match the correct files

find /var/qmail/mailnames -mtime +182 -type f -regex ".*\.$HOSTNAME,.*S=.*,[a-yA-Y]*" ! -name "*\.gz*"