1

When a spammer uses his botnet of thousands of zombie PCs to spam at random nonexistent addresses @example.com, at a very high rate of speed, postfix exhausts our VPS provider's resource limits trying to deal with it. Specifically, it exhausts the number of non-TCP sockets, which is limited to 900 by the VPS. Am running postfix 2.3.3-2.1.el5_2 on CentOS 5 on a Virtuozzo linux VPS.

/var/log/maillog says:
Feb 23 06:26:22 postfix/smtpd[3938]: warning: connect #1 to subsystem private/proxymap: Cannot allocate memory
Feb 23 06:26:22 postfix/smtpd[3936]: fatal: socket: Cannot allocate memory
Feb 23 06:26:48 postfix/qmgr[17702]: fatal: socket: Cannot allocate memory

Firewalling would be kind of tough because of the thousands of IPs involved in the dictionary attack.

The VPS provider suggested tuning the following parameters, but did not give suggestions on what to set them to:

max_idle = 100s (default)
max_use = 100 (default)

I found another person having the same problem with postfix and spammer dictionary attacks:

http://forums.vpslink.com/linux/394-you-hitting-socket-resource-limits-2.html#post5241

He changed:

default_process_limit from 100 (default) to 10

... which solved the problem but introduced a performance penalty.

I am unsure exactly which parameter should be safely tuned here, even after skimming postfix.org/TUNING%5FREADME.html Can any postfix expert help?

ane
  • 171
  • 1
  • 4
  • 16

2 Answers2

1

Sadly, as Postfix follows the process driven model, high memory usage under load is one of it's side effect. You could try this

From /etc/postfix/master.cf

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       n       -       5       smtpd

In the maxproc column, you can replace the - with a smaller number to limit the number of concurrent smtpd processes, this should provide some gate on the amount of incoming mail.

Another alternative would be to look at fail2ban which can be configured to parse /var/log/maillog and raise iptables blocks for addresses that send a large amount of undeliverable mail.

Dave Cheney
  • 18,567
  • 8
  • 49
  • 56
  • Thank you Dave. Any suggestions on what the smaller number should be? This is a fairly active server with several hundred accounts and almost 60 different domains, so I'm not quite comfortable with only 5 processes yet. Right now, for example, there are 12 smtpd processes and no dictionary attacks going on (thankfully). On a similar server, there are 27 smtpd processes. Should I just double the higher number and make it say, 54 possible processes and fine-tune it based on trial & error? Or is there a more scientific way to do this? – ane Feb 23 '10 at 21:09
  • Whoa, 12? How may cores do you have on this beast? What that means is you can have 12 *simultaneous* connections - which potentially means 12 very busy CPUs, if you have virus scanners attached to postfix. If you only have a quad or dual processor server, you'll want to trim that number up a bit, say 8 or 4. Yeah, you won't be able to get a connection while the spam storm is raging, but it will rate-limit the spammer too...and prevent your box from keeling over. I guess a better question is: how many users would realistically connect to send *all at once*? Take that number, and add 2. – Avery Payne Feb 24 '10 at 02:48
  • Ana - from your initial post it appeared your problem was memory usage, not cpu usage, because you were working in a small VPS. Don't forget that Postfix works on a per process model, which will fork one `smtpd` for the _life_ of the smtp transation. That is from `EHLO` to closing the connection once the message has gone through your spam filters, etc. If you need more performance per mb of ram, Postfix probably isn't the server for you. – Dave Cheney Feb 24 '10 at 05:02
  • The specific problem is the number of non-TCP sockets eaten by postfix during dictionary attacks (thousands) is more than the provider provides (max: 900). It appears that the more smtpd processes exist, the more non-TCP sockets are used (?) When postfix hits this limit, it gives a memory allocation error and stops accepting new connections. There is actually plenty of memory free (over 1GB) but the non-TCP sockets are maxed out. – ane Feb 24 '10 at 16:18
  • Finally got around to trying Dave's advice. After decreasing it to 5, I now receive the following: Mar 26 12:00:03 postfix/master[23946]: warning: service "smtp" (25) has reached its process limit "5": new clients may experience noticeable delays Mar 26 12:00:03 postfix/master[23946]: warning: to avoid this condition, increase the process count in master.cf or reduce the service time per client So I set it back to the default. Guess some experimentation is needed to figure out the correct balance of max processes. Still not sure how maxproc affects non-TCP sockets. – ane Mar 26 '10 at 16:04
0

You might want to consider adding something similar to:

smtpd_recipient_restrictions =
        permit_auth_destination,
        permit_mynetworks,
        reject_unauth_destination,
        reject_unlisted_recipient

...to your main.cf file, which should cause postfix to dump some of the connections as soon as it figures out there isn't a user to deliver to. The magic is in reject_unlisted_recipient, which will cause rejections for users that are not valid on your system when used with local_recipient_maps. By doing so, it should take some pressure off by reducing the amount of processing going on, as each dropped connection frees up precious resources to deal with the next connection. Yes, the spammer will be able to determine what addresses are invalid, but better that the spammer wastes their time sending (and being rejected, wasting their bandwidth) than yours (fending off the attack).

Avery Payne
  • 14,536
  • 1
  • 51
  • 88
  • Thanks Avery. That's the default setting but I moved it higher in the smtpd_recipient_restrictions list. After monitoring, the number of non-TCP sockets remains about the same, so I'm not sure this particular setting has much effect on non-TCP sockets. – ane Feb 24 '10 at 16:48