View what file started a process in linux

Question

One of my servers started sending spam in heavy amounts. The server is using ISPConfig 3 and one of the web users most likely has a security breach.

I have identified the user and one file somehow starts a perl script that sends the spam.

However i cannot find the file(s) that is affected since most likely everything is ran in the memory.

Apparently that file creates somehow a service with the name "mail" that runs on the server and binds itself to the main ip addresses.

Below i'll post some information regarding my findings since I'm out of ideas on how to track the problem.

ps aux | grep mail
web24     5101  1.8  0.3  35580  7944 ?        Ss   10:45   6:01 mail
web24     5102  1.6  0.2  34628  5160 ?        Ss   10:45   5:28 mail

netstat -nalp | grep :25
tcp        0      1 x.x.x.x:50774    194.54.81.164:25        SYN_SENT    5102/mail
tcp        0      1 x.x.x.x:47849    194.54.81.163:25        SYN_SENT    5101/mail

Most likely the script is triggered manually since it starts randomly after being closed.

I would like to know how to trace which file started that process (most likely I will have to analize the memory dumps).

After running lsof on the PID I get the following:

lsof -p 31459
COMMAND   PID  USER   FD   TYPE  DEVICE SIZE/OFF     NODE NAME
perl    31459 web24  cwd    DIR    0,34     4096 18735409 /
perl    31459 web24  rtd    DIR    0,34     4096 18735409 /
perl    31459 web24  txt    REG    0,34     7360 19644668 /usr/bin/perl
perl    31459 web24  mem    REG   253,0          19644668 /usr/bin/perl (path dev=0,34)
perl    31459 web24  mem    REG   253,0          20621055 /usr/lib/perl/5.10.1/auto/File/Glob/Glob.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          20620896 /usr/lib/perl/5.10.1/auto/POSIX/POSIX.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          20621014 /usr/lib/perl/5.10.1/auto/Fcntl/Fcntl.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          20627464 /usr/lib/perl/5.10.1/auto/Socket/Socket.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          20621012 /usr/lib/perl/5.10.1/auto/IO/IO.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          19628140 /lib/libcrypt-2.11.3.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          19628224 /lib/libc-2.11.3.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          19628216 /lib/libpthread-2.11.3.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          19628144 /lib/libm-2.11.3.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          19628220 /lib/libdl-2.11.3.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0          19644608 /usr/lib/libperl.so.5.10.1 (path dev=0,34)
perl    31459 web24  mem    REG   253,0          19628132 /lib/ld-2.11.3.so (path dev=0,34)
perl    31459 web24  mem    REG   253,0           5857281 /usr/lib/locale/locale-archive (path dev=0,34)
perl    31459 web24    0r   CHR     1,3      0t0 19220558 /dev/null
perl    31459 web24    1w   CHR     1,3      0t0 19220558 /dev/null
perl    31459 web24    2w   CHR     1,3      0t0 19220558 /dev/null
perl    31459 web24    3u  IPv4 5906021      0t0      TCP my.hostname.ro:46504->164.81.54.194.static.server.ua:smtp (SYN_SENT)
perl    31459 web24    4w  FIFO     0,8      0t0  3061201 pipe
perl    31459 web24    5r  FIFO     0,8      0t0  3061202 pipe
perl    31459 web24    6w  FIFO     0,8      0t0  3061548 pipe
perl    31459 web24    7r  FIFO     0,8      0t0  3061549 pipe

I should mention that there is no suspicious cronjob and I am sure the script is ran manually since now it has the name "httpd" instead of "mail" as I previously mentioned.

I could solve the issue very easy, just close the client's account and move on, but since this kind of security breach is my first I cannot fix I would like to solve it for future reference.

score 0 · Answer 1 · answered Jan 14 '16 at 15:37

0

You can have a look at what lsof is showing.

lsof | grep mail

Check what files are open by the process:

lsof -p PID

and maybe strace can also show you something:

strace -p PID

I would also check if there are not suspicious looking scripts added to cron.

answered Jan 14 '16 at 15:37

morholt

1

Thanks, but I've already tried that and no luck. However the attacker changed the name of the running script to httpd. I've updated my question. – Alex Jan 18 '16 at 11:56

score 0 · Answer 2 · answered Jan 15 '16 at 11:25

I agree with running lsof -p 5101 and lsof -p 5102 to see what files are actually opened by that process. You would also be able to get more information by running cat /proc/5101/environ and cat /proc/5102/environ. Look for the SCRIPT_FILENAME variable and this will show your location. I've seen many times that cat /proc/PID/environ will not return any output - this is because the attackers forge the environment. So, it's safe to say that lsof -p PID is your safest bet in finding the location. Once you find the location of the script, before removing it you should run stat script.pl or whatever the name is to see WHEN the script has been placed on your server. That timestamp will be helpful in finding the entry in the log to see exactly how it happened.

Thanks for the advice, however the environ is blank as you suspected. I've updated my question with the output of lsof. — Alex, Jan 18 '16 at 12:05
So, the script might reside in the home directory of web24. You might want to also get a list of files that have been placed in the last month you can run `find /home/web24 -type f -mtime -30` This has helped me a lot of times when dealing with hacked accounts. Make sure you actually use the homedir of web24 - I used /home/web24 assuming that is indeed your home directory. — Mugurel, Jan 18 '16 at 12:26

View what file started a process in linux

2 Answers2