1

As a bit of background, it seems that our server got infected with something and is being used to open a ton of TCP connections over a large range if IPs. I'm halfway through trying to track down how our server got infected now; my tale of woe has been outlined at 398639 for anyone who wants some additional information.

The current issue is I've found an Apache command "con.shs" that is routinely taking up 100% of our CPU (it's definitely possible it's related to our server compromise).

My question is if anyone knew what "con.shs" is and why it's running at 100%? No Google search has returned anything that might help.

We're running Centos 5.7 Final, and Apache 2.2.3 (with PHP and MySQL).

Community
  • 1
dKen
  • 193
  • 9

1 Answers1

1

con.shs could just be a random name the malware picks. Have you tried inspecting the process?

Use pgrep con.shs to find the list of PIDs and inspect the /proc/<pid>/ directory - look at details such as exe (what the executable is - unfortunately, they delete it sometimes) and perhaps cwd (what the working directory of the script is - in my experience, they don't bother running it from somewhere like /tmp). Other files in there will be useful too, such as cmdline.

This should help you track it down, see what it's doing, and prevent it from coming back.

Jay
  • 6,544
  • 25
  • 34
  • Wow, fantastic info. I've done this and tracked down everything you've said. `exe` file is in `/dev/shm`, and `cmdline` contains `/bin/bashs190.480`, so it definitely looks suspicious. My problem now is what to do with that information. Do I delete those files? Is it possible to find out how it was installed? – dKen Jun 14 '12 at 13:22
  • You can do `ps auxwf` (or `pstree`) to see what forked it, and close that hole. `/dev/shm` is a temporary filesystem, it'll get deleted on reboot anyway, so you need to stop it coming back. – Jay Jun 14 '12 at 13:55
  • Thanks Jay, looks like `/usr/sbin/httpd` forked the process, which isn't overly useful. Next step I guess is to find out which files were edited when the compromise occurred and see if I can track down the offending file (if there is one), before I blast everything away and rebuild? Have you got any reading suggestions for tracing issues like this? – dKen Jun 14 '12 at 14:16
  • It's difficult to say without direct access. What's your permissions on files? Are they group writeable by whatever user httpd is running as (e.g. apache, nobody)? If not, they probably can't write anywhere to the filesystem except `/tmp` or `/dev/shm`. If this is the case, they are probably actively exploiting some kind of bug that allows them to do remote code execution *but* not be able to write this data to disk anywhere persistent. Take a look at your `access_log` file and see if there's anything obvious (perhaps even `error_log` if their attack wasn't blind and used that fact). – Jay Jun 14 '12 at 15:07
  • Great response, thanks Jay. Will go away and do some more research and patch all the software I have. FYI, it was a trojan that I tracked down with your answers/comments; thanks. – dKen Jun 14 '12 at 15:36
  • No problem :-) Best of luck! – Jay Jun 14 '12 at 15:53