2

I am working with a compromised Ubuntu 8.04 Plesk 9.5.4 server. It seems that a script on the server is continuously doing reverse lookups to random IPs on the Internet.

I first spotted it during by using top and then noticed flashes of this coming up continuously: sh -c host -W 1 '198.204.241.10'

I wrote a this script to interrogate ps every 1 second to see how frequently this script happens:

#!/bin/bash
while :
do
    ps -ef | egrep -i "sh -c host"
    sleep 1
done

The results are that this script runs often, every few seconds:

www-data 17762  8332  1 10:07 ?        00:00:00 sh -c host -W 1 '59.58.139.134'
www-data 17772  8332  1 10:07 ?        00:00:00 sh -c host -W 1 '59.58.139.134'
www-data 17879 17869  0 10:07 ?        00:00:00 sh -c host -W 1 '198.204.241.10'
www-data 17879 17869  1 10:07 ?        00:00:00 sh -c host -W 1 '198.204.241.10'
www-data 17879 17869  0 10:07 ?        00:00:00 sh -c host -W 1 '198.204.241.10'
root     18031 17756  0 10:07 pts/2    00:00:00 egrep -i sh -c host
www-data 18078 16704  0 10:07 ?        00:00:00 sh -c host -W 1 '59.58.139.134'
www-data 18125 17996  0 10:07 ?        00:00:00 sh -c host -W 1 '91.124.51.65'
root     18131 17756  0 10:07 pts/2    00:00:00 egrep -i sh -c host
www-data 18137 17869  0 10:07 ?        00:00:00 sh -c host -W 1 '198.204.241.10'
www-data 18137 17869  1 10:07 ?        00:00:00 sh -c host -W 1 '198.204.241.10'

My theory is if I can see who is launching the sh process or form where it's launched I can isolate the problem further.

Can somebody please guide me using netstat or ps to identify from where sh is being run?

I might get many suggestions that the OS is out of date and so the Plesk, but please bear in mind there are some very concrete reasons why this server is running legacy software. My question is aimed at a advanced Linux systems administrators who have in depth experience with security compromises and using netstat and ps to get to the bottom of it.

UPDATE 26 Oct 14:01 SAST

The command grep -lr 'sh -c host' left in the comments left by @ramruma helped me to quickly find the corresponding website running the script. As it turns out the server is not compromised. What is instead happening is there is a forum in that location produced by Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC that does a reverse lookup every time someone accesses the board.

It also turns out that the forum is unmaintained and severely spammed and why we initially thought it's compromised is because many of the reverse lookups pointed to foreign IPs known for spamming. Goodness only knows how a website can execute this command, but for the time being I am going to close the issue as harmless.

With regards to the blanket replies I received about "restore from backup" I came to ServerFault to help isolate the issue and I made it clear in my initial request. Sometimes restoring from backup is impractical or even impossible. It's better to figure out the issue or find the compromise in case it might arise in the future.

Eugene van der Merwe
  • 435
  • 5
  • 9
  • 21
  • Since it's running as `www-data` user, how about this `grep -lr 'sh -c host' /path/to/the/document/root`? – quanta Oct 26 '13 at 08:24
  • 2
    also `pwdx $(pgrep -f host)` and pstree on the parent pid, and lsof to see which files the parent process has open. – ramruma Oct 26 '13 at 08:32
  • it'd be more efficient to have solved this first and then described. See my profile if interested. – poige Oct 26 '13 at 09:58
  • @ramruma, you command has assisted me in finding the exact website strait away where this script is run. I would like to mark your answer as correct, but since you only commented this is impossible. – Eugene van der Merwe Oct 26 '13 at 11:59

2 Answers2

1

While it looks like you've figured out the cause of this problem already, dumping the process table with pstree and a few args would have helped.

Replace your ps command in the loop with:
pstree -apu www-data >> getyousomespam.txt

You'd only want to run this for about 30 seconds due to the wall of text it would generate. This will display all processes owned by www-data, their ancestor processes, all uid transitions along the way (-u), commandline args that you can search on (-a), and PIDs (-p).

Obviously this information would be completely bunk if you've been compromised, but if you haven't (as seems to be the case here), this would provide the paper trail you were looking for...or at least a good start.

Edit:
pstree doesn't concern itself with originating terminals, but if you can isolate things to a persistent ancestor process, having the PID would make it easy to finish the job from there.

Andrew B
  • 32,588
  • 12
  • 93
  • 131
0

Stop. You're going about this the wrong way. Your server is compromised, you need to disconnect it and restore from a known-good backup. See also How do I deal with a compromised server?

Community
  • 1
Dennis Kaarsemaker
  • 19,277
  • 2
  • 44
  • 70