What could cause Mailman to drop messages after moderator approval?

Question

I run a medium-sized Mailman system that recently developed a problem where any messages that pass through moderation disappear instead of being delivered to the mailing list. This is affecting every one of our mailing lists.

Moderation fails when performed on a separate webserver

The Mailman environment is split across two servers, front-end and back-end. The back-end server handles Postfix and the Mailman qrunners, while the front-end server hosts Apache and the Mailman CGI scripts for moderating lists. The two servers share an NFS mount between them that includes all the shared Mailman data.

All normaly mail flow is working correctly, but when a list moderator logs into the web frontend and approves a message, it disappears without a trace.

1. Postfix smtpd receives the incoming message over SMTP, then
2. Postfix smtpd delivers the message to /usr/lib/mailman/mail/mailman.
3. Mailman marks writes to vette logfile (backend server) that message is held for approval.
4. List moderator uses CGI web interface to mark the message as approved.
5. Mailman writes an entry to vette logfile (on frontend server) saying held message approved.

At this point, the .pck file related to the held message disappears, but nothing is delivered, and no further log entries are created.

Moderation succeeds with web interface on the main Mailman server

Although we don't normally run the Mailman web interface on the back-end server (to reduce attack surface), I got it running for testing purposes. When we use the Mailman web interface on the backend server, the message gets delivered normally and we see these log entries.

1. smtp logfile updated with number of recipients and time for completion
2. post logfile updated with list name, message ID, and "success".

Background

The problem started after migrating the Mailman environment to new servers. It didn't crop up on it's own, it's most likely a result of some configuration error that we haven't caught yet. We're using:

• Scientific Linux 6.3 on both servers
• Python 2.6.6 on both servers
• Mailman 2.1.12 installed from OS packages on both servers
• selinux in Permissive mode on backend server
• selinux in Enforcing mode on frontend (web) server, but no log entries with type=AVC are being recorded. Furthermore, using setenforce 0 doesn't fix the problem.

I found one related post on the Mailman users list, but no solution was provided.

Answer 1

When using more than one server for Mailman, every server needs to have access to the queue directories on shared storage. That's it.

Understanding where moderated messages go

1. If a message is held for moderation, it is moved into $DATA_DIR and the message ID is appended to $LIST_DATA_DIR/listname/pending.pck.
2. The Mailman web interface looks in pending.pck to find messages that are held for moderation. When a moderator approves the held message, it is moved into the $INQUEUE_DIR folder.

Which data needs to be shared?

This is what I recommend if you have a separate server handling the Mailman web frontend.

MUST be on shared storage

• queue_dir, inqueue_dir, outqueue_dir, cmdqueue_dir, bouncequeue_dir, newsqueue_dir, archqueue_dir, shuntqueue_dir, virginqueue_dir, badqueue_dir, retryqueue_dir, maildir_dir Your queue files need to be accessible by any server that runs Mailman tasks, including the frontend web server.

• DATA_DIR, LIST_DATA_DIR In addition to mail queues, you also need to share all of the list configuration files and held message files.

• PUBLIC_ARCHIVE_FILE_DIR, PRIVATE_ARCHIVE_FILE_DIR If you're using list archives, then you'll also need to share the archive directories.

SHOULD be on shared storage

• LOCK_DIR, PID_DIR, PIDFILE I'm not completely sure, but it seems like the locks and pidfiles should be located on shared storage so that if something bad happens to the qrunner server, it will be evident that the processes ended abnormally.

• SITE_PW_FILE, LISTCREATOR_PW_FILE You probably want your password files to be on shared storage so that you can be sure your master list password works no matter which server you're on.

• CONFIG_DIR If you're using MTA=Postfix, Mailman will automatically create the aliases file in CONFIG_DIR. Since any machine with Mailman can be used to create or delete lists, each machine should also be able to update a shared aliases file correctly. (Caveat Emptor: Sometimes you want to configure Mailman slightly differently on each machine, which could be difficult with a shared CONFIG_DIR.)

MAY be on shared storage

• LOG_DIR Depending on your preferences, you can either keep these directories local or put them on shared storage. I like having all my logs in one central place that is being backed up, so that I still have old logs available after migrating the processes to a new server.

• TEMPLATE_DIR If you have customized your Mailman templates at all (eg. bounce messages) then you probably want to have those on shared storage too.

• SPAM_DIR I don't know what SPAM_DIR is actually used for, but it's recommended that all of the variable files are on shared storage so I'm including that here.

Local storage only

• WRAPPER_DIR, BIN_DIR, SCRIPTS_DIR, MESSAGES_DIR It's a very good idea to keep binaries and scripts local, so that you can take advantage of packages provided by your operating system for upgrades without worrying about keeping the shared binaries in sync. Mailman seems to be very particular about having exactly the same version running on any server that is involved with the shared storage.

(Edited on 2013-09-04) The following guidance was provided by Mark Sapiro on the Mailman-Users listserv.

My advice would be for standard GNU Mailman to share all the mutable data which is everything in var_prefix which is all of the directories archives/, data/, lists/, locks/, logs/, qfiles/ and spam/, however you have a Scientific Linux (Red Hat derivative) package, so see the FAQ at http://wiki.list.org/x/KYCB for how these map to your install.

There is a FAQ at http://wiki.list.org/x/wgB0 that addresses this somewhat. It might need some additions. Do see all the links.

(Edited on 2013-09-04) Actually config dir should be shared not local.

Answer 2

Mark Sapiro posted on Mailman-users 03-Dec-2013:

http://www.mail-archive.com/mailman-users@python.org/msg63365.html

"Mailman 2.1.12 is not compatible with Python 2.6+. This was fixed in 2.1.13."

This particular issue was fixed by this patch:

(I am not including the patch here, as the formatting would get corrupted.)