Sendmail is taking huge amount of CPU

Question

My dedicated server is slowing down since 2 days. I checked the RPM and the CPU usage has gone from less than 10% to more than 50%. Rebooting the server didn't help.

So, I tried top command:

top - 19:10:24 up 57 min,  1 user,  load average: 2.37, 2.71, 2.79
Tasks: 256 total,   3 running, 253 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.6 us,  2.1 sy,  0.0 ni, 75.4 id,  6.8 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:  66005120 total,  9632784 used, 56372336 free,   214996 buffers
KiB Swap:  1046520 total,        0 used,  1046520 free,  6309748 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
  14662 web1      20   0  314m  47m  12m R  49.9  0.1   2:23.22 php-cgi
  9755 web1      20   0  316m  49m  13m S  43.6  0.1   7:38.70 php-cgi
  14683 web1      20   0  318m  49m  13m R  39.9  0.1   2:59.74 php-cgi
  5992 root      20   0  305m 9988 4736 S   3.3  0.0   5:45.75 PassengerAgent
  5286 mysql     20   0  627m 226m 7984 S   2.7  0.4   1:42.51 mysqld 
  9 root      20   0     0    0    0 S   0.3  0.0   0:02.03 rcu_sched
  5539 root      20   0 20516 3832 1988 S   0.3  0.0   0:06.58 vlogger (access 
  17710 www-data  20   0  375m  16m 2588 S   0.3  0.0   0:00.03 apache2

Which shows that the user web1 is using a lot of cpu with PHP. So I tried

ps aux | grep web1

here's the output (seems the sendmail is creating an issue):

root@nameserver:/var/www/clients/client0/web1/tmp# ps aux | grep web1
web1      9755 18.0  0.0 324372 50528 ?        R    18:27   7:43 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1      9907 19.2  0.1 615992 71176 ?        Sl   18:28   8:08 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1      9913 18.4  0.0 328628 54516 ?        R    18:28   7:49 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1     10976 15.9  0.0 331224 57272 ?        R    18:32   6:00 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1     11002 18.9  0.0 333584 58956 ?        R    18:32   7:07 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1     14662 13.4  0.0 323332 49068 ?        R    18:52   2:27 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1     14671 18.3  0.0 329400 54976 ?        R    18:52   3:19 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1     14672 18.1  0.0 328632 54456 ?        S    18:52   3:17 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com
web1     14683 16.8  0.0 326208 51676 ?        S    18:52   3:01 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client0/web1/web:/var/www/clients/client0/web1/private:/var/www/clients/client0/web1/tmp:/var/www/example.com/web:/srv/www/example.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin -d upload_tmp_dir=/var/www/clients/client0/web1/tmp -d session.save_path=/var/www/clients/client0/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -fwebmaster@example.com

How to get things back to normal.

Could you please post copy/pastes of your logs instead of unreadable screenshots? — EEAA, Jun 09 '15 at 17:24
I changed to logs. sendmail log (/var/log/mail.info, /var/log/mail.log, /var/log/messages ) are not showing strange things. It's pretty normal actually. — Louis XIV, Jun 09 '15 at 17:38
Tentatively, it looks like an exploited or exploitable PHP script is being used to send bulk email. You should stop sendmail immediately and disable said scripts. — Craig Watson, Jun 09 '15 at 20:15
Thank you for your conclusion, but I did think about it from the beginning, it seems pretty obvious actually that it is one of the possibilities... But, how can I be sure about it? I won't throw my whole server on some supposition. How can I monitor sendmail activity since there is nothing in the logs? Seriously, give some help instead of giving your opinion. — Louis XIV, Jun 10 '15 at 01:12
If you don't know how to verify this, you should pay someone to come in and help you out. (This is not snarkiness; this is experience talking.) — Jenny D, Jun 11 '15 at 14:20

Sendmail is taking huge amount of CPU

0 Answers0