0

I'm looking at a server that appears to have been compromised via a bug in a wordpress plugin and is now sending spam out.

This post seems to give a very good report of exactly what I'm seeing.

Wordpress hack

So I've taken these steps...

list the users running stuff in crontab.

for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done | more

Bingo, loads of scripts for my USER that all look suspect. This list is this line a lot..

* * * * * /home/tart/public_html/wp-includes/SimplePie/Content/1.sh

So I edit the crontab for this user and remove them.

crontab -u tart -e

Now I run this to see if span is still being sent...

tail -n 10 /var/log/exim_mainlog -F

ARRGH, its still alive.

Ok, let me look at TOP

30596 root      20   0 71728 5864 3680 S  0.3  0.1   0:00.01 exim
30602 root      20   0 71728 5864 3680 S  0.3  0.1   0:00.01 exim
30608 root      20   0 71724 5860 3680 S  0.3  0.1   0:00.01 exim
30609 mailnull  20   0 71736 5244 3088 D  0.3  0.1   0:00.01 exim

Now I'm not running mail on this server, just a webserver. So exim is not being used by me.

So let me stop exim.

/etc/init.d/exim status

/etc/init.d/exim stop

/etc/init.d/exim status

exim now report as stopped

So I check the log again (tail -n 10 /var/log/exim_mainlog -F)

Stuff is still appearing in the log.

Let me check the status again...

/etc/init.d/exim status
exim (pid 3169 3159) is running...

ARGGG.

Also I can see loads of spam messages in the queue to go out.... exim -bp

So my question is this.

How do I stop this?!?

I have "WHM Accelerated2" panel, and when I check "Service Manager" I can see "exim Mail server" and "exim Mail server (but another port)" are disabled.

Note: this is "CENTOS 6.5 x86_64 standard – ds-71494 WHM 11.46.0 (build 17)"

Extra info:

This is the list of cron jobs for root.

root
0 6 * * * /usr/local/cpanel/scripts/exim_tidydb > /dev/null 2>&1
30 5 * * * /usr/local/cpanel/scripts/optimize_eximstats > /dev/null 2>&1
2,58 * * * * /usr/local/bandmin/bandmin
0 0 * * * /usr/local/bandmin/ipaddrmap
53 22 * * * /usr/local/cpanel/scripts/upcp --cron
0 1 * * * /usr/local/cpanel/scripts/cpbackup
0 2 * * * /usr/local/cpanel/bin/backup
35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check
45 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_mailman_cache && /usr/local/cpanel/scripts/update_mailman_cache
30 */4 * * * /usr/bin/test -x /usr/local/cpanel/scripts/update_db_cache && /usr/local/cpanel/scripts/update_db_cache
45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel/bin/optimizefs
30 */2 * * * /usr/local/cpanel/bin/mysqluserstore >/dev/null 2>&1
15 */2 * * * /usr/local/cpanel/bin/dbindex >/dev/null 2>&1
15 */6 * * * /usr/local/cpanel/scripts/autorepair recoverymgmt >/dev/null 2>&1
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
23 22 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
5,20,35,50 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1

UPDATE

I'm 98% sure I know how attack took place, and I'm 98% sure the hole they used is closed. (I can't be sure another hole doesn't exist though)

Taking it offline: this isn't that easy since it's a hosted server.

Rebuild: yes, give me a few days to sort this, but right now I want to stop the spam and understand a bit more about how exim is being activated.

The thing that worries me is that exim is running as root (as shown by top). My root password I changed last night, and was/is of the format of 123$£!23#asd2" (not a simple password)

Looking at the outbound count, I see...

root@ds-111110 [~]# grep '<=' /var/log/exim_mainlog | awk '{print $5}' | grep \@
   5227 tart@ds-11110.com
 619519 root@ds-11110.com

So it seems root is main culprit of the spam, again makes me worry.

Also note, there is no Customer/sensitive data on this server, making a rebuild easy-ish.

HopelessN00b
  • 53,795
  • 33
  • 135
  • 209
jeff porter
  • 119
  • 1
  • 8
  • 2
    `server that appears to have been compromised` - Turn it off. Reinstall. See: http://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server – Zoredache Nov 26 '14 at 20:07
  • My experience with hacked PHP sites is that it's often not really necessary to reinstall, because it's quite clear what a hack might have involved, especially one that was just a bot that found a way to send spam (not a directed attack). In fact, if you reinstall the same way, especially if you tar your PHP sites and redeploy them, it will be compromised the same way. BTW, enable PHP mail log, so you can see what site is the culprit. Sometimes sending spam is possible just by calling a URL on your site. Also, some process may be running in the background, called from cron. Reboot might help. – Halfgaar Nov 26 '14 at 20:24
  • 1
    @Halfgaar: It is very wise to **always** reinstall after a compromise unless you know for **certain** what the attacker did, which is usually not the case. Finding and closing the attack vector should be obvious. – Sven Nov 26 '14 at 20:27
  • 2
    @Halfgaar My experience with most people who ask this type of question is that they don't have good logging, backups or the ability to actually discover the full impact of a compromise. So the safe generalized approach is often the best. – Zoredache Nov 26 '14 at 20:30
  • I do have a backup of this site, if that backup contains the hack or not I'm not sure. For now its stopped sending spam, tomorrow I'll look at the backup. – jeff porter Nov 26 '14 at 20:53

1 Answers1

0

Though I very strongly think you should nuke any rebuild any compromised system, here is some information for you to address the symptom.

On an *nix system usually your Mail Transfer Agent (MTA) will include a binary that emulates the the sendmail interface. Any program on your system can call this to add deliver a message. Stopping the MTA Daemon doesn't stop this binary from accepting messages, and depending on the MTA this binary will even deliver the message to the Internet without the MTA daemon running.

Some programs can also make outgoing tcp/25 connections, completely bypassing the MTA altogether.

In your case it is likely that your scripts are simply calling sendmail. Stopping the daemon doesn't do anything useful for stopping email. A simple temporary solution might be to add a iptables rules that completely blocks output tcp/25, tcp/465, tcp/587 connections. These are the well known SMTP ports. You could also change the permissions on /usr/sbin/sendmail, and make it so that it isn't executable (chmod 0644 /usr/sbin/sendmail).

This is not a fix for your problem, this is just a temporary plug while you make a final backup and collect any data before you re-build your system.

You need to find the actual root of your problem, and need to fix that.

Zoredache
  • 130,897
  • 41
  • 276
  • 420
  • I've chmod'd sendmail, and so far after 5 minutes nothing has appeared in exim_mainlog log. Thanks for this tip. – jeff porter Nov 26 '14 at 20:40
  • 1
    I'll add one comment to clarify/correct something above. When you use the init script to stop exim, it stops the listening process only. The init script is designed so that if you do have existing exim PIDs in the process of receiving (inbound connections) or sending emails (running the queue), it leaves those running so the transfer can finish. If you run '/etc/init.d/exim stop' a second time, it will kill all those processes that were left running. The rest of the recommendations are accurate. – Todd Lyons Nov 26 '14 at 20:57