Identify where and how background process are started and where they comes from

Question

First the short story: I need to migrate a server (applications, configurations and so on) and I have no clue about what is in there, no docs, people at charge just abandon and didn't leave any information so it's a kind of black box or black hole. My task, move what is in that server to a new instance and know how things in there works. The problem there are some background process running (see ps -ax output below):

ps -ax

Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
  PID TTY      STAT   TIME COMMAND
  ...
  841 ?        Ss    13:42 python /usr/local/bin/pdoInstaller/
  848 ?        Ss     0:04 php /usr/local/bin/pdoneVendorBroker/vendorBroker.php
  950 ?        Ssl   13:00 /usr/bin/mongod --config /etc/mongodb.conf
  013 ?        S      0:00 CRON
 1014 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 1015 ?        Ssl    1:02 /usr/sbin/mysqld
 1016 ?        S      0:02 /usr/bin/php rss_article_loader.php
 1065 ?        Ssl    0:29 /usr/sbin/nova-agent -q -p /var/run/nova-agent.pid -o /var/log/nova-agent.log -l info /usr/share/nova-agent/nova-agent.py
 1219 ?        S      0:01 /usr/lib/erlang/erts-5.10.3/bin/epmd -daemon
 1222 ?        S      0:00 CRON
 1223 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 1224 ?        S      0:01 /usr/bin/php rss_article_loader.php
 1506 ?        S      0:00 /bin/sh /usr/sbin/rabbitmq-server
 1517 ?        Sl    15:59 /usr/lib/erlang/erts-5.10.3/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@pdone-db-qa -boot /v
 1728 ?        S      0:00 CRON
 1729 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 1730 ?        S      0:00 /usr/bin/php rss_article_loader.php
 3137 ?        Ss     0:04 php /usr/local/bin/shareEventHandler/shareEventHandler.php
 3165 ?        Ss     0:04 php /usr/local/bin/repToolBroker/repToolBroker.php
 3180 ?        Ss     0:04 php /usr/local/bin/pdoneLoginProctor/loginProctor.php
 3201 ?        Ss     0:04 php /usr/local/bin/messageBroker/messageBroker.php
 3230 ?        Ss     0:04 php /usr/local/bin/emailBroker/emailBroker.php
 3250 ?        Ss     0:04 php /usr/local/bin/edetailBroker/edetailBroker.php
 3270 ?        Ss     0:04 php /usr/local/bin/cmeBroker/cmeBroker.php
 3921 ?        S      0:00 CRON
 3922 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 3923 ?        S      0:03 /usr/bin/php rss_article_loader.php
 4395 ?        S      0:00 CRON
 4396 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 4397 ?        S      0:02 /usr/bin/php rss_article_loader.php
 4498 ?        S      0:00 CRON
 4499 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 4500 ?        S      0:01 /usr/bin/php rss_article_loader.php
 5781 ?        S      0:00 CRON
 5782 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 5783 ?        S      0:04 /usr/bin/php rss_article_loader.php
 7242 ?        S      0:00 CRON
 7243 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 7244 ?        S      0:03 /usr/bin/php rss_article_loader.php
 7575 ?        S      0:00 CRON
 7576 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 7577 ?        S      0:02 /usr/bin/php rss_article_loader.php
 7705 ?        S      0:00 CRON
 7706 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 7707 ?        S      0:01 /usr/bin/php rss_article_loader.php
 9368 ?        S      0:00 CRON
 9369 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
 9370 ?        S      0:04 /usr/bin/php rss_article_loader.php
10450 ?        S      0:00 CRON
10451 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
10452 ?        S      0:03 /usr/bin/php rss_article_loader.php
10771 ?        S      0:00 CRON
10772 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
10773 ?        S      0:02 /usr/bin/php rss_article_loader.php
10884 ?        S      0:00 CRON
10885 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
10886 ?        S      0:01 /usr/bin/php rss_article_loader.php
12947 ?        S      0:00 CRON
12949 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
12951 ?        S      0:04 /usr/bin/php rss_article_loader.php
13573 ?        S      0:00 CRON
13574 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
13575 ?        S      0:03 /usr/bin/php rss_article_loader.php
13963 ?        S      0:00 CRON
13964 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
13965 ?        S      0:01 /usr/bin/php rss_article_loader.php
14157 ?        S      0:00 CRON
14158 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
14159 ?        S      0:00 /usr/bin/php rss_article_loader.php
16083 ?        S      0:00 CRON
16084 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
16085 ?        S      0:04 /usr/bin/php rss_article_loader.php
17089 ?        S      0:00 CRON
17090 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
17091 ?        S      0:03 /usr/bin/php rss_article_loader.php
17103 ?        S      0:00 CRON
17104 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
17105 ?        S      0:01 /usr/bin/php rss_article_loader.php
17553 ?        S      0:00 CRON
17554 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
17555 ?        S      0:00 /usr/bin/php rss_article_loader.php
19227 ?        S      0:00 CRON
19228 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
19229 ?        S      0:04 /usr/bin/php rss_article_loader.php
20318 ?        S      0:00 CRON
20319 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
20320 ?        S      0:01 /usr/bin/php rss_article_loader.php
20375 ?        S      0:00 CRON
20376 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
20377 ?        S      0:02 /usr/bin/php rss_article_loader.php
20722 ?        S      0:00 CRON
20723 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
20724 ?        S      0:00 /usr/bin/php rss_article_loader.php
22324 ?        S      0:00 CRON
22325 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
22326 ?        S      0:03 /usr/bin/php rss_article_loader.php
23549 ?        S      0:00 CRON
23550 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
23551 ?        S      0:01 /usr/bin/php rss_article_loader.php
23643 ?        S      0:00 CRON
23644 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
23645 ?        S      0:02 /usr/bin/php rss_article_loader.php
23945 ?        S      0:00 CRON
23946 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
23947 ?        S      0:00 /usr/bin/php rss_article_loader.php
25875 ?        S      0:00 CRON
25876 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
25877 ?        S      0:03 /usr/bin/php rss_article_loader.php
26840 ?        S      0:00 CRON
26841 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
26842 ?        S      0:02 /usr/bin/php rss_article_loader.php
27223 ?        S      0:00 CRON
27225 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
27227 ?        S      0:01 /usr/bin/php rss_article_loader.php
27538 ?        S      0:00 CRON
27539 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
27540 ?        S      0:00 /usr/bin/php rss_article_loader.php
29374 ?        S      0:00 CRON
29375 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
29376 ?        S      0:03 /usr/bin/php rss_article_loader.php
30232 ?        S      0:00 CRON
30233 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
30234 ?        S      0:02 /usr/bin/php rss_article_loader.php
30444 ?        S      0:00 CRON
30445 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
30446 ?        S      0:01 /usr/bin/php rss_article_loader.php
30682 ?        S      0:00 /usr/sbin/apache2 -k start
30683 ?        S      0:00 /usr/sbin/apache2 -k start
30848 ?        S      0:00 CRON
30849 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
30850 ?        S      0:00 /usr/bin/php rss_article_loader.php
32692 ?        S      0:00 CRON
32693 ?        Ss     0:00 /bin/sh -c cd /var/www/pdone/ipad_v2/rpc; /usr/bin/php rss_article_loader.php
32694 ?        S      0:03 /usr/bin/php rss_article_loader.php

Some are running through CRON I have that ones identified, the easy path, already but others like the ones started by PHP meaning for example:

 3137 ?        Ss     0:04 php /usr/local/bin/shareEventHandler/shareEventHandler.php
 3165 ?        Ss     0:04 php /usr/local/bin/repToolBroker/repToolBroker.php
 3180 ?        Ss     0:04 php /usr/local/bin/pdoneLoginProctor/loginProctor.php

I can't get where that comes from and I need to know where and how they are started in order to setup the same on the new server, can any gives some ideas in how to attack this problem? The only thing I know at this moment is RabbitMQ use that script to deliver messages and do some tasks. I need to identify where and how background process are started and where they comes from. Original server is Ubuntu the new one is CentOS is not a problem but just FYI, any can give me some help or ideas?

Update

I have found a cmeBroker.conf file at /usr/local/bin/cmeBroker directory and so on for the rest of them, this is the content of that file:

description "PDone cmeBroker"

start on runlevel [234]
stop on runlevel [0156]

respawn
exec php /usr/local/bin/cmeBroker/cmeBroker.php
post-start script
    PID=`status cmeBroker | egrep -oi '([0-9]+)$' | head -n1`
    echo $PID > /var/run/cmeBroker.pid
end script

post-stop script
    rm -f /var/run/cmeBroker.pid
end script

So this is a clue for now, but who is the responsible for start or read this file?

Update 2

As suggested by @ivan I got installed auditd on the Ubuntu server and I've ran this command:

auditctl -w /usr/local/bin/repToolBroker/repToolBroker.php -p rwxa

Should I restart the server or just wait and find using:

ausearch -f /usr/local/bin/repToolBroker/repToolBroker.php

for who accessed or changed the file? That could show me who is the responsible for start those files and execute them in background?

Update 3

Using @julian-sivertsen suggestion for search the filesystem for text strings that match the file name I thanks to another help I got from here I run this command:

sudo grep -r 'shareEventHandler.php\|repToolBroker.php\|loginProctor.php\|messageBroker.php\|emailBroker.php\|edetailBroker.php\|cmeBroker.php' .

And that allow me to find where the files was called. This is the output for the command above:

./init/emailBroker.conf:exec php /usr/local/bin/emailBroker/emailBroker.php
./init/cmeBroker.conf:exec php /usr/local/bin/cmeBroker/cmeBroker.php
./init/pdoneLoginProctor.conf:exec php /usr/local/bin/pdoneLoginProctor/loginProctor.php
./init/edetailBroker.conf:exec php /usr/local/bin/edetailBroker/edetailBroker.php
./init/messageBroker.conf:exec php /usr/local/bin/messageBroker/messageBroker.php
grep: ./init/veevaBroker.conf: No such file or directory
./init/shareEventHandler.conf:exec php /usr/local/bin/shareEventHandler/shareEventHandler.php
./init/repToolBroker.conf:exec php /usr/local/bin/repToolBroker/repToolBroker.php

The .conf file are symlink to the file I've found before and this is where all happens. Now I have one question remaining, what the init folder is for? Is my first time dealing with this and I'll like to know and learn.

Any?

That's going to be a nightmare. It's clear the previous admin was ... inexperienced ... and the developers of whatever is running on there were also ... inexperienced. To put it as nicely as I can. I suggest [running for the hills](http://www.urbandictionary.com/define.php?term=run+for+the+hills). — Michael Hampton, Mar 27 '15 at 17:44
@MichaelHampton sorry for my ignorance, what "running from the hills" means? And yes it's a nightmare I don't know where to start from — ReynierPM, Mar 27 '15 at 17:47

Julian Sivertsen · Accepted Answer · 2015-03-27T21:39:02.557

3

Start by investigating the relations between parent/child processes using a command such as ps -AF --forest. The Parent Process ID (PPID) will either be the process that spawned the process in question, or 1 if it forked or became orphaned. The --forest switch to PS displays a graphical representation of this relationship.

Deamons are usually started with scripts in /etc/rc5.d/, try looking there. Also look at the files for cron tasks. The locations and setup for these differs depending on Linux distribution, consult the distribution documentation for details. For the upstart init system, it's possible to list all daemons using the service --status-all command.

If those comes up empty, you can try searching the filesystem for text strings that match the commands used to start the processes. For example grep "rss_article_loader\\.php" -r /usr /etc will show all files in /usr and /etc containing the string 'rss_article_loader.php'. Note that grep searches with regular expressions and not simple text strings.

edited Mar 27 '15 at 21:39

answered Mar 27 '15 at 19:50

Julian Sivertsen

66
4

Hi, I have run the command as you suggest but still don't know where the latest script are executed from. Take a look to the [output](http://pastie.org/10058265) and notice the orphans PHP scripts at the end `shareEventHandler, repToolBroker, loginProctor, messageBroker, emailBroker, edetailBroker, cmeBroker` those are the ones I need to find their executor or trigger, have any other idea than this one? It helps but doesn't give me what I'm looking for – ReynierPM Mar 27 '15 at 21:24
Also see my edition, perhaps this can help – ReynierPM Mar 27 '15 at 21:31
1

The file you found looks like an upstart config file. Try running `service --status-all` and see if they are listed there. – Julian Sivertsen Mar 27 '15 at 21:41
Nop, they are not listed, so ... – ReynierPM Mar 27 '15 at 22:10
Can you give me some advice around `Update 3` doubt regarding `init` folder? – ReynierPM Mar 28 '15 at 05:15
If those links are in `/etc/init` then it's probably managed by upstart. Look at the manual page for upstart and init for documentation of how it works. – Julian Sivertsen Mar 28 '15 at 12:39

score 0 · Answer 2 · edited Mar 17 '17 at 13:14

0

I had luck identifying culprits with the Linux auditing subsystem.

Linux audit files to see who made changes to a file

linux - Monitoring system calls - Information Security Stack Exchange

edited Mar 17 '17 at 13:14

Community

1

answered Mar 27 '15 at 21:49

ivan_pozdeev

352
4
13

Fine, I have installed `auditd` cause the OS is Ubuntu. I have read the post you leave me but this is complete new for me. Can you improve your answer adding, as an example, how you would audit one of those files? – ReynierPM Mar 27 '15 at 22:14
See Update2 for details about what I did so far – ReynierPM Mar 27 '15 at 22:20

Identify where and how background process are started and where they comes from

2 Answers2