3

When I run ps aux | grep /usr/local/apache/bin/httpd I get the following output.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data  9837  0.0  0.0  23112  1360 ?        S    Oct15   0:00 /usr/local/apache/bin/httpd -DSSL
www-data  9841  0.0  0.0  23112  1568 ?        S    Oct15   0:16 /usr/local/apache/bin/httpd -DSSL
www-data 29178  0.0  0.0  23112  1064 ?        S    Oct04   1:51 /usr/local/apache/bin/httpd -DSSL

What I find interesting though is that /usr/local/apache/bin/httpd doesn't exist. There is no /usr/local/apache/bin/ directory at all.

sudo cat /proc/9837/cmdline returns /usr/local/apache/bin/httpd -DSSL

When I run /proc/9837$ sudo /usr/local/apache/bin/httpd -DSSL I get:

/usr/local/apache/bin/httpd: command not found

When I run sudo ls -l /proc/9837/exe I get:

lrwxrwxrwx 1 www-data www-data 0 2012-10-17 02:06 /proc/9837/exe -> /usr/bin/perl

My question is, why is this happening? Shouldn't /proc/<pid>/cmdline and /proc/<pid>/exe be related? If this is not normal, what steps can I take to find out what caused it?

These processes remain even after running sudo /etc/init.d/apache2 stop

For reference, I'm running Ubuntu Ubuntu 10.04.4 LTS running Apache2 from the default apt repository.

sudo ls -l /proc/9837/fd outputs

lr-x------ 1 www-data www-data 64 2012-10-17 02:47 0 -> /dev/null
l-wx------ 1 www-data www-data 64 2012-10-17 02:47 1 -> pipe:[37796710]
l-wx------ 1 www-data www-data 64 2012-10-17 02:47 2 -> /var/log/apache2/error.log
lrwx------ 1 www-data www-data 64 2012-10-17 02:47 3 -> socket:[37796725]
lr-x------ 1 www-data www-data 64 2012-10-17 02:47 4 -> pipe:[40055427]

stat /proc/9837/root returns

  File: `/proc/9837/root' -> `/'
  Size: 0           Blocks: 0          IO Block: 1024   symbolic link
Device: 3h/3d   Inode: 49853155    Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2012-10-17 02:07:00.240782014 -0400
Modify: 2012-10-17 02:06:43.860777313 -0400
Change: 2012-10-17 02:06:43.860777313 -0400
Citricguy
  • 166
  • 1
  • 1
  • 9
  • Does `/var/log/apache2/error.log` actually exists? Is it possible that you have a perl program running as a wrapper to a chrooted Apache environment? You could check for a possible `chroot()` environment by inspecting `/proc/[pid]/root`. (by the way, [this is a good reference](http://www.kernel.org/doc/man-pages/online/pages/man5/proc.5.html)) – Alexander Janssen Oct 17 '12 at 06:57
  • /proc/$PID/root looks like a symlink to `/`. `/var/log/apache2/error.log` is valid as well. – Citricguy Oct 17 '12 at 07:10

2 Answers2

2

Smells fishy. Programs can override their $0. The program pretends to be Apache, but it's perl in fact.

Go to /proc/$PID/fd and have a look what files it's got opened, maybe this gives you a clue what's going on.

quanta
  • 51,413
  • 19
  • 159
  • 217
Alexander Janssen
  • 2,607
  • 16
  • 21
  • I added the output to the end of the question. Is it possible to retrieve the original command elsewhere if /proc/$PID/cmdline has been overwritten? – Citricguy Oct 17 '12 at 06:58
  • @Citricguy In fact, I'm already looking for that. I *thought* I've seen that before, but somehow I can't find it right know. Will keep looking. – Alexander Janssen Oct 17 '12 at 07:00
0

you said that you had installed apache2 from default repository. can you run a consistency check based on that package to check what is missing ?

It seems that someone / something / somehow deleted some of the apache files. But apache had been started before that, so it still remains in memory

Nikolaidis Fotis
  • 2,032
  • 11
  • 13
  • The original/system apache2 is matches whats in Ubuntu's repositories and can restart without issue. This apache2 process loads from `/usr/sbin/apache2` as it should. – Citricguy Oct 17 '12 at 08:50
  • up to which depth you have the directories ? /usr/local/apache ? /usr/local/apache/bin/ ? – Nikolaidis Fotis Oct 17 '12 at 09:04
  • Only as high as `/usr/local/`, but I don't think this is where Apache is located in Ubuntu 10.04 (Lucid). I believe the binaries live in `/usr/sbin` but I'll verify as this is a good point. – Citricguy Oct 17 '12 at 09:51