Server OOM crash nightly caused by multiple malicious POST requests?

Question

So it appears that every night at around midnight some server from China attempts to access my drupal site. From the looks of the logs it keeps making the same request every 61 seconds (so as to avoid being flagged by the firewall probably). This request is a POST request on the user registration page and whatever the request seems to tie up the Apache process, so every minute a new Apache process is spawned until the server runs out of memory and swaps itself into a coma. I've of course blocked the IP on the firewall but I want to get to the bottom of why the request locks up Apache. What's the best way to go about debugging this?

Here's a look at the Apache status log:

Srv PID Acc M   CPU SS  Req Conn    Child   Slot    Client  VHost   Request
0-0 7331    0/8/65  W   8.84    687 0   0.0 0.08    0.35    192.74.226.108  ---.org POST /user/register HTTP/1.1
1-0 6409    0/33/33 W   13.58   206 0   0.0 8.16    8.16    94.228.34.211   ---.org GET /clinic-design/forum/all/jweddingtonMD HTTP/1.1
2-0 6410    0/3/3   W   6.19    648 0   0.0 0.00    0.00    192.74.226.108  --.org  POST /user/register HTTP/1.1
3-0 6411    0/27/27 W   6.83    254 0   0.0 0.11    0.11    157.55.34.25    --.org  GET /chd/membership/individual-members HTTP/1.1
4-0 6412    0/25/25 W   13.34   201 0   0.0 0.17    0.17    192.74.226.108  --.org  POST /user/register HTTP/1.1
5-0 6417    0/3/3   W   8.10    566 0   0.0 0.03    0.03    192.74.226.108  --.org  POST /user/register HTTP/1.1
6-0 7531    0/0/19  W   6.05    323 0   0.0 0.00    0.06    192.74.226.108  --.org  POST /user/register HTTP/1.1
7-0 6428    0/19/19 W   11.50   223 0   0.0 0.44    0.44    192.74.226.108  --.org  POST /user/register HTTP/1.1
8-0 7447    0/6/13  W   1.98    444 0   0.0 0.04    0.04    192.74.226.108  --.org  POST /user/register HTTP/1.1
9-0 6842    0/38/38 W   13.33   262 0   0.0 0.38    0.38    192.74.226.108  --.org  POST /user/register HTTP/1.1
10-0    7499    0/0/14  W   0.00    405 0   0.0 0.00    0.05    192.74.226.108  --.org  POST /user/register HTTP/1.1
11-0    6845    0/22/22 W   11.11   505 0   0.0 0.23    0.23    192.74.226.108  --.org  POST /user/register HTTP/1.1
12-0    6953    1/64/64 W   14.08   930 0   0.0 0.83    0.83    192.74.226.108  --.org  POST /user/register HTTP/1.1
13-0    6954    0/10/10 W   3.09    282 0   0.0 0.47    0.47    192.74.226.108  --.org  POST /user/register HTTP/1.1
14-0    7502    0/0/74  W   0.00    384 0   0.0 0.00    0.98    192.74.226.108  --.org  POST /user/register HTTP/1.1
15-0    7191    0/52/113    W   26.77   466 0   0.0 0.53    1.06    192.74.226.108  --.org  POST /user/register HTTP/1.1
16-0    7010    0/77/77 W   11.89   869 0   0.0 0.58    0.58    192.74.226.108  --.org  POST /user/register HTTP/1.1
17-0    7023    0/67/67 W   8.52    892 0   0.0 0.80    0.80    192.74.226.108  --.org  POST /user/register HTTP/1.1
18-0    7358    0/0/37  W   7.63    809 0   0.0 0.00    0.56    192.74.226.108  --.org  POST /user/register HTTP/1.1
19-0    7437    0/17/79 W   10.23   161 0   0.0 0.16    4.08    157.55.34.25    --.org  GET /--/membership/individual-members HTTP/1.0
20-0    7100    0/74/74 W   6.51    831 0   0.0 0.79    0.79    192.74.226.108  --.org  POST /user/register HTTP/1.1
21-0    7192    0/44/47 W   5.94    626 0   0.0 1.40    1.40    192.74.226.108  --.org  POST /user/register HTTP/1.1
22-0    7126    0/37/37 W   10.65   770 0   0.0 3.15    3.15    192.74.226.108  --.org  POST /user/register HTTP/1.1
23-0    7183    1/20/20 W   5.27    952 0   0.0 0.03    0.03    192.74.226.108  --.org  POST /user/register HTTP/1.1
24-0    7503    0/4/34  W   3.14    206 0   0.0 0.00    0.20    66.249.73.106   --.org  GET /--/membership/student-members?order=city&sort=desc&last_n
25-0    7193    0/35/35 W   14.07   748 0   0.0 1.04    1.04    192.74.226.108  --.org  POST /user/register HTTP/1.1
26-0    7566    0/0/15  W   1.67    194 0   0.0 0.00    0.02    94.228.34.211   --.org  GET /clinic-design/forum?page=0%2C5 HTTP/1.1
27-0    7400    0/18/19 W   8.04    527 0   0.0 0.08    0.08    192.74.226.108  --.org  POST /user/register HTTP/1.1
28-0    7401    0/0/0   W   0.00    709 0   0.0 0.00    0.00    192.74.226.108  --.org  POST /user/register HTTP/1.1
29-0    7402    0/2/2   W   0.00    588 0   0.0 0.00    0.00    192.74.226.108  --.org  POST /user/register HTTP/1.1
30-0    7569    0/2/6   W   0.00    141 0   0.0 0.00    0.00    192.74.226.108  --.org  POST /user/register HTTP/1.1
31-0    7465    0/5/6   W   3.15    345 0   0.0 0.05    0.05    192.74.226.108  --.org  POST /user/register HTTP/1.1
32-0    7466    0/8/8   W   5.56    163 0   0.0 0.14    0.14    192.74.226.108  --.org  POST /user/register HTTP/1.1
33-0    7574    0/2/2   W   0.02    123 0   0.0 0.00    0.00    46.227.71.215   --.org  GET /aggregator/sources/1?page=1 HTTP/1.1
34-0    7577    0/7/7   W   2.10    41  0   0.0 0.00    0.00    192.74.226.108  --.org  POST /user/register HTTP/1.1
35-0    7581    0/0/0   W   0.00    168 0   0.0 0.00    0.00    113.212.69.10   --.org  GET /?q=user HTTP/1.1
36-0    7586    0/0/0   W   0.00    139 0   0.0 0.00    0.00    67.195.115.123  ---.org GET /--/conferences-events/calendar-events/environmental-stand
37-0    7587    0/0/0   W   0.00    138 0   0.0 0.00    0.00    146.251.88.193  --.org  GET /edac HTTP/1.1
38-0    7616    0/0/0   W   0.00    135 0   0.0 0.00    0.00    67.227.237.76   --.org  POST /sites/all/modules/civicrm/bin/civimail.cronjob.php HTTP/1
39-0    7617    0/1/1   W   0.00    102 0   0.0 0.01    0.01    192.74.226.108  --.org  POST /user/register HTTP/1.1
40-0    7618    0/0/0   W   0.00    134 0   0.0 0.00    0.00    157.55.32.142   --.org  GET /--/programs/awards-recognition/changemaker-award/2003-cha
41-0    7628    0/0/0   W   0.00    106 0   0.0 0.00    0.00    146.251.88.193  --.org  GET /edac HTTP/1.1
42-0    7629    0/0/0   W   0.00    105 0   0.0 0.00    0.00    157.55.32.142   --.org  GET /--/about/meet-team/ellen-taylor-aia-mba-edac HTTP/1.1
43-0    7641    0/5/5   _   1.92    9   0   0.0 0.00    0.00    66.249.73.75    store.--.org    GET /publications.html?SID=259b106f3c06e307ec810593e4b15edf&lim
44-0    7642    0/1/1   W   0.00    75  0   0.0 0.00    0.00    157.55.32.142   --.org  GET /--/resources/webinars?page=1 HTTP/1.1
45-0    7644    0/0/0   W   0.00    80  0   0.0 0.00    0.00    192.74.226.108  --.org  POST /user/register HTTP/1.1
46-0    7647    0/1/1   W   0.01    62  0   0.0 0.00    0.00    146.251.88.193  --.org  GET /edac HTTP/1.1
47-0    7660    0/0/0   W   0.00    73  0   0.0 0.00    0.00    67.195.115.123  --.org  GET /--/conferences-events/calendar-events/environmental-stand
48-0    7661    0/2/2   W   0.00    15  0   0.0 0.00    0.00    157.55.32.142   --.org  GET /--/programs/awards-recognition/changemaker-award/2003-cha
49-0    7662    0/0/0   W   0.00    45  0   0.0 0.00    0.00    157.55.32.142   --.org  GET /node/146/nurture-collegiate-healthcare-design-compet?page=
50-0    7663    0/0/0   W   0.00    53  0   0.0 0.00    0.00    67.195.115.123  --.org  GET /--/conferences-events/calendar-events/environmental-stand
51-0    7667    0/0/0   W   0.00    32  0   0.0 0.00    0.00    67.195.115.123  --.org  GET /--/conferences-events/calendar-events/environmental-stand
52-0    7669    0/0/0   W   0.00    26  0   0.0 0.00    0.00    66.249.73.106   --.org  GET /clinic-design/design-process/pre-design/plan-program-d HTT
53-0    7670    0/0/0   W   0.00    25  0   0.0 0.00    0.00    199.21.99.99    --.org  GET /resources/pubs/ HTTP/1.1
54-0    7671    0/0/0   W   0.00    19  0   0.0 0.00    0.00    192.74.226.108  --.org  POST /user/register HTTP/1.1
55-0    7673    0/1/1   W   0.00    9   0   0.0 0.00    0.00    67.195.115.123  --.org  GET /--/conferences-events/calendar-events/environmental-stand
56-0    7675    0/0/0   W   0.00    0   0   0.0 0.00    0.00    127.0.0.1   host.--.org GET /whm-server-status HTTP/1.1

score 0 · Answer 1 · answered Mar 09 '13 at 23:02

0

Modify the registration page to dump $_POST to a log file. Then you will be able to see what they are sending.

answered Mar 09 '13 at 23:02

longneck

23,082
4
52
86

score 0 · Answer 2 · answered Mar 10 '13 at 04:25

I've seen the same bots, and I've dealt with it by writing my own fail2ban jail for them.

This configuration is tuned to block for a day, after six attempts in an hour. After being in production for a couple of months, it has yet to block a legitimate registration attempt. It does, however, send a lot of mail, so you might want to tune that...

You may need to tweak the regex a bit if your log files are in any way unusual (e.g. not in the Apache combined style).

/etc/fail2ban/jail.conf contains, in part:

[drupal-user-register]
enabled  = true
filter   = drupal-user-register
action   = iptables-multiport[name=DrupalRegBots, port="http,https"]
           sendmail-buffered[name=DrupalRegBots, lines=5, dest=webmaster@example.com]
logpath  = /var/log/nginx/example.com-access.log
           /var/log/nginx/example.com-ssl-access.log
bantime  = 86400
findtime = 3600
maxretry = 6

/etc/fail2ban/filter.d/drupal-user-register.conf contains:

# Fail2Ban configuration file
#
# Author: Michael Hampton
#
# $Revision$
#

[Definition]

# Option:  failregex
# Notes.:  regex to Drupal match user registration page attempts
# Values:  TEXT
#
failregex = ^<HOST> .*(GET|POST) /user/register .*$

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

score 0 · Answer 3 · answered Mar 11 '13 at 19:59

The W state is sending reply. It's very easy to get an Apache server stuck in this state essentially forever; just fail to ACK the packets it's sending in a timely manner, and keep your receive window laughably small.

Consider running a reverse proxy like Varnish in front, so that something other than Apache can handle the nasty business of dealing with real clients.

Server OOM crash nightly caused by multiple malicious POST requests?

3 Answers3