so recently i've noticed lots of email related process suddenly popup in my server like so
Mem: 7895616k total, 1599756k used, 6295860k free, 83044k buffers
Swap: 8388540k total, 625920k used, 7762620k free, 643768k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
190985 mysql 20 0 1342m 182m 4488 S 5.2 2.4 64:42.18 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin
143308 mailnull 20 0 74300 7644 3700 S 1.3 0.1 0:00.04 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
143347 mailnull 20 0 74296 7640 3700 S 1.3 0.1 0:00.04 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
143348 mailnull 20 0 74296 7644 3700 S 1.3 0.1 0:00.04 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
143358 mailnull 20 0 74304 7624 3676 D 1.3 0.1 0:00.04 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
143515 mailnull 20 0 74316 8212 4228 S 1.3 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIl-000bHn-9s
143516 mailnull 20 0 74316 8212 4228 S 1.3 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIl-000bI2-BK
143526 mailnull 20 0 74320 8212 4228 S 1.3 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIl-000bHp-Bi
143537 mailnull 20 0 74320 8212 4228 S 1.3 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIk-000bI4-LE
143544 mailnull 20 0 74296 7644 3700 S 1.3 0.1 0:00.04 /usr/sbin/exim -odi -t -oem -oi -f <> -E1bnBIl-000bHn-9s
143553 mailnull 20 0 74316 8160 4188 D 1.3 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIm-000bKe-O0
143571 root 20 0 74340 8064 4068 D 1.3 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIk-000bI5-L5
143572 root 20 0 74344 8068 4068 D 1.3 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIl-000bI3-Au
143349 mailnull 20 0 74300 7644 3700 S 1.0 0.1 0:00.03 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
143357 mailnull 20 0 74296 7644 3700 S 1.0 0.1 0:00.03 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t -f root
143479 mailnull 20 0 74320 8212 4228 S 1.0 0.1 0:00.04 /usr/sbin/exim -odi -Mc 1bnBIk-000bFw-Mj
143508 mailnull 20 0 74296 7644 3700 S 1.0 0.1 0:00.03 /usr/sbin/exim -odi -t -oem -oi -f <> -E1bnBIk-000bFw-Mj
143541 mailnull 20 0 74300 7644 3700 S 1.0 0.1 0:00.03 /usr/sbin/exim -odi -t -oem -oi -f <> -E1bnBIl-000bI2-BK
143554 mailnull 20 0 74300 7624 3676 D 1.0 0.1 0:00.03 /usr/sbin/exim -odi -t -oem -oi -f <> -E1bnBIl-000bHp-Bi
143557 mailnull 20 0 74320 8216 4228 S 1.0 0.1 0:00.03 /usr/sbin/exim -odi -Mc 1bnBIl-000bHQ-9W
143564 mailnull 20 0 74300 7628 3676 D 1.0 0.1 0:00.03 /usr/sbin/exim -odi -t -oem -oi -f <> -E1bnBIk-000bI4-LE
143570 root 20 0 74344 8072 4068 D 1.0 0.1 0:00.03 /usr/sbin/exim -odi -Mc 1bnBIk-000bID-KA
in my server i have lots of cronjobs running but email report is disabled in all of them and i rarely use my SMTP to send email in my crontab for root user i have
SHELL=/bin/bash
PATH=/usr/local/bin/:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=
HOME=/
* * * * * root /usr/local/bin/proc /home/rdsc/public_html/reader.js par1
* * * * * root /usr/local/bin/proc /home/rdsc/public_html/reader.js par2
so i've searched and everybody suggested tht i should look into /var/log/mailog which i did
Sep 23 00:35:11 centos-67-64-minimal dovecot: pop3-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<5ttsBB89vdN/AAAB>
Sep 23 00:35:11 centos-67-64-minimal dovecot: lmtp(138733): Connect from local
Sep 23 00:35:11 centos-67-64-minimal dovecot: lmtp(138733): Disconnect from local: Successful quit
Sep 23 00:35:12 centos-67-64-minimal dovecot: imap-login: Login: user=<__cpanel__service__auth__imap__pw7m8ugksshamq9pv3jbshxkm1pt38wa5kgxntqgfeqe62...>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=138746, secured, session=<akZ+BB89Y5l/AAAB>
Sep 23 00:35:12 centos-67-64-minimal dovecot: imap(__cpanel__service__auth__imap__pw7m8ugksshamq9pv3jbshxkm1pt38wa5kgxntqgfeqe62znxi28rx0dh43qogfr): Logged out in=11, out=462, bytes=11/462
Sep 23 00:37:13 centos-67-64-minimal spamd[96881]: prefork: killing failed child 138709 fd=8 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:37:13 centos-67-64-minimal spamd[96881]: prefork: killed child 138709
Sep 23 00:37:13 centos-67-64-minimal spamd[96881]: prefork: killing failed child 138708 fd=7 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:37:13 centos-67-64-minimal spamd[96881]: prefork: killed child 138708
Sep 23 00:40:09 centos-67-64-minimal spamd[96881]: zoom: able to use 997/998 'body_0' compiled rules (99.899%)
Sep 23 00:40:11 centos-67-64-minimal spamd[96881]: spamd: server started on IO::Socket::INET6 [_]:783, IO::Socket::INET [#177_]:783 (running version 3.4.1)
Sep 23 00:40:12 centos-67-64-minimal spamd[96881]: spamd: server pid: 96881
Sep 23 00:40:12 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 141754
Sep 23 00:40:12 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 141756
Sep 23 00:40:12 centos-67-64-minimal spamd[141754]: spamd: connection from localhost [::1]:36734 to port 783, fd 5
Sep 23 00:40:12 centos-67-64-minimal spamd[96881]: prefork: child states: BI
Sep 23 00:40:12 centos-67-64-minimal spamd[96881]: prefork: child states: II
Sep 23 00:40:12 centos-67-64-minimal dovecot: pop3-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<+dplFh898dV/AAAB>
Sep 23 00:40:13 centos-67-64-minimal dovecot: lmtp(141776): Connect from local
Sep 23 00:40:13 centos-67-64-minimal dovecot: lmtp(141776): Disconnect from local: Successful quit
Sep 23 00:40:13 centos-67-64-minimal dovecot: imap-login: Login: user=<__cpanel__service__auth__imap__nu9f0ujsobi4ynuhjveddgqztv59i7nsabfgjh4xckias6...>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=141781, secured, session=</Tp1Fh89mJt/AAAB>
Sep 23 00:40:13 centos-67-64-minimal dovecot: imap(__cpanel__service__auth__imap__nu9f0ujsobi4ynuhjveddgqztv59i7nsabfgjh4xckias6lpr1muswf_c_avlf7a): Logged out in=11, out=462, bytes=11/462
Sep 23 00:42:14 centos-67-64-minimal spamd[96881]: prefork: killing failed child 141754 fd=7 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:42:14 centos-67-64-minimal spamd[96881]: prefork: killed child 141754
Sep 23 00:42:14 centos-67-64-minimal spamd[96881]: prefork: killing failed child 141756 fd=8 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:42:14 centos-67-64-minimal spamd[96881]: prefork: killed child 141756
Sep 23 00:42:14 centos-67-64-minimal spamd[96881]: spamd: handled cleanup of child pid [141756] due to SIGCHLD: interrupted, signal 2 (0002)
Sep 23 00:45:09 centos-67-64-minimal spamd[96881]: zoom: able to use 997/998 'body_0' compiled rules (99.899%)
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: spamd: server started on IO::Socket::INET6 [_]:783, IO::Socket::INET [#177_]:783 (running version 3.4.1)
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: spamd: server pid: 96881
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 144782
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 144783
Sep 23 00:45:11 centos-67-64-minimal spamd[144782]: spamd: connection from localhost [::1]:37207 to port 783, fd 5
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: prefork: child states: BS
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 144785
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: prefork: adjust: 0 idle children less than 1 minimum idle children. Increasing spamd children: 144785 started.
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: prefork: child states: BIS
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: prefork: child states: BII
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: prefork: child states: III
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: prefork: adjust: 3 idle children more than 2 maximum idle children. Decreasing spamd children: 144785 killed.
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: prefork: child states: IIK
Sep 23 00:45:11 centos-67-64-minimal spamd[96881]: spamd: handled cleanup of child pid [144785] due to SIGCHLD: interrupted, signal 2 (0002)
Sep 23 00:45:11 centos-67-64-minimal dovecot: pop3-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<FHc6KB89pdd/AAAB>
Sep 23 00:45:12 centos-67-64-minimal dovecot: lmtp(144804): Connect from local
Sep 23 00:45:12 centos-67-64-minimal dovecot: lmtp(144804): Disconnect from local: Successful quit
Sep 23 00:45:12 centos-67-64-minimal dovecot: imap-login: Login: user=<__cpanel__service__auth__imap__4m_db2dzy8221unq7wvfponlfavw2sbj22gbkimaiykibs...>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=144810, secured, session=<WiVLKB89S51/AAAB>
Sep 23 00:45:12 centos-67-64-minimal dovecot: imap(__cpanel__service__auth__imap__4m_db2dzy8221unq7wvfponlfavw2sbj22gbkimaiykibsnoje20hqkfyznicx9s): Logged out in=11, out=462, bytes=11/462
Sep 23 00:47:13 centos-67-64-minimal spamd[96881]: prefork: killing failed child 144782 fd=7 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:47:13 centos-67-64-minimal spamd[96881]: prefork: killed child 144782
Sep 23 00:47:13 centos-67-64-minimal spamd[96881]: prefork: killing failed child 144783 fd=8 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:47:13 centos-67-64-minimal spamd[96881]: prefork: killed child 144783
Sep 23 00:47:13 centos-67-64-minimal spamd[96881]: spamd: handled cleanup of child pid [144783] due to SIGCHLD: interrupted, signal 2 (0002)
Sep 23 00:50:09 centos-67-64-minimal spamd[96881]: zoom: able to use 997/998 'body_0' compiled rules (99.899%)
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: spamd: server started on IO::Socket::INET6 [_]:783, IO::Socket::INET [#177_]:783 (running version 3.4.1)
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: spamd: server pid: 96881
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 147686
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 147687
Sep 23 00:50:10 centos-67-64-minimal spamd[147686]: spamd: connection from localhost [::1]:37673 to port 783, fd 5
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: prefork: child states: BS
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: spamd: server successfully spawned child process, pid 147689
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: prefork: adjust: 0 idle children less than 1 minimum idle children. Increasing spamd children: 147689 started.
Sep 23 00:50:10 centos-67-64-minimal spamd[96881]: prefork: child states: BII
Sep 23 00:50:11 centos-67-64-minimal spamd[96881]: prefork: child states: III
Sep 23 00:50:11 centos-67-64-minimal spamd[96881]: prefork: adjust: 3 idle children more than 2 maximum idle children. Decreasing spamd children: 147689 killed.
Sep 23 00:50:11 centos-67-64-minimal spamd[96881]: prefork: child states: IIK
Sep 23 00:50:11 centos-67-64-minimal spamd[96881]: spamd: handled cleanup of child pid [147689] due to SIGCHLD: interrupted, signal 2 (0002)
Sep 23 00:50:11 centos-67-64-minimal dovecot: pop3-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<JCMTOh89dNl/AAAB>
Sep 23 00:50:12 centos-67-64-minimal dovecot: lmtp(147708): Connect from local
Sep 23 00:50:12 centos-67-64-minimal dovecot: lmtp(147708): Disconnect from local: Successful quit
Sep 23 00:50:12 centos-67-64-minimal dovecot: imap-login: Login: user=<__cpanel__service__auth__imap__zbgyfocbcalr9crjm0mia55dyfablygrk7sb0p4mhud8ov...>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=147713, secured, session=<V08jOh89G59/AAAB>
Sep 23 00:50:12 centos-67-64-minimal dovecot: imap(__cpanel__service__auth__imap__zbgyfocbcalr9crjm0mia55dyfablygrk7sb0p4mhud8ovr24lwemk6ng2x4yyfx): Logged out in=11, out=462, bytes=11/462
Sep 23 00:52:13 centos-67-64-minimal spamd[96881]: prefork: killing failed child 147686 fd=7 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:52:13 centos-67-64-minimal spamd[96881]: prefork: killed child 147686
Sep 23 00:52:13 centos-67-64-minimal spamd[96881]: prefork: killing failed child 147687 fd=8 at /usr/local/cpanel/3rdparty/perl/522/lib64/perl5/cpanel_lib/Mail/SpamAssassin/SpamdForkScaling.pm line 172.
Sep 23 00:52:13 centos-67-64-minimal spamd[96881]: prefork: killed child 147687
i dont get any of this !
is it possible that someone using my server to send spam ? is there a simpler log around ?
or maybe i can setup some kind of log to find out which account/username sending email and which emails are receiving them ?
after Rayan suggestion i've located the exim log and its not pretty
2016-09-23 02:01:18 1bmpr7-003lCt-Cr Unfrozen by errmsg timer
2016-09-23 02:01:18 1bmpr7-003lCt-Cr ** root@centos-67-64-minimal R=fail_remote_domains: The mail server could not deliver mail to root@centos-67-64-minimal. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.
2016-09-23 02:01:18 1bmpr7-003lCt-Cr root@CentOS-67-64-minimal: error ignored
2016-09-23 02:01:18 1bmpr7-003lCt-Cr Completed
2016-09-23 02:01:18 1bmrm7-004Hym-MF Message is frozen
2016-09-23 02:01:18 1bmuL7-000b9P-79 Message is frozen
2016-09-23 02:01:18 1bn1x7-002bnM-CU Message is frozen
2016-09-23 02:01:18 1bmwc7-0019B8-SX Message is frozen
2016-09-23 02:01:18 1bmtF7-000K08-5W Message is frozen
2016-09-23 02:01:18 1bmvJ7-000qjJ-FU Message is frozen
2016-09-23 02:01:18 1bmtF7-000K00-3q Message is frozen
2016-09-23 02:01:18 1bn1T7-002XT2-QW Message is frozen
2016-09-23 02:01:18 1bn0V7-002Dzn-KI Message is frozen
2016-09-23 02:01:18 1bmpr7-003lCZ-4f Unfrozen by errmsg timer
2016-09-23 02:01:18 1bmpr7-003lCZ-4f ** root@centos-67-64-minimal R=fail_remote_domains: The mail server could not deliver mail to root@centos-67-64-minimal. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.
2016-09-23 02:01:18 1bmpr7-003lCZ-4f root@CentOS-67-64-minimal: error ignored
2016-09-23 02:01:18 1bmpr7-003lCZ-4f Completed
2016-09-23 02:01:18 1bmsH7-0004ia-IE Message is frozen
2016-09-23 02:01:18 1bn4r7-003N2Z-HO Message is frozen
2016-09-23 02:01:18 1bmrm7-004Hyw-Ov Message is frozen
2016-09-23 02:01:19 1bmps7-003lN9-5A Unfrozen by errmsg timer
2016-09-23 02:01:19 1bmps7-003lN9-5A ** root@centos-67-64-minimal R=fail_remote_domains: The mail server could not deliver mail to root@centos-67-64-minimal. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.
2016-09-23 02:01:19 1bmps7-003lN9-5A root@CentOS-67-64-minimal: error ignored
2016-09-23 02:01:19 1bmps7-003lN9-5A Completed
2016-09-23 02:01:19 1bmz37-001pjQ-Ck Message is frozen
2016-09-23 02:01:19 1bn167-002Mv1-Re Message is frozen
2016-09-23 02:01:22 1bmpIJ-003fQs-Sq Unfrozen by errmsg timer
2016-09-23 02:01:22 1bmpIJ-003fQs-Sq ** root@centos-67-64-minimal R=fail_remote_domains: The mail server could not deliver mail to root@centos-67-64-minimal. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.
2016-09-23 02:01:22 1bmpIJ-003fQs-Sq root@CentOS-67-64-minimal: error ignored
2016-09-23 02:01:22 1bmpIJ-003fQs-Sq Completed
2016-09-23 02:01:22 1bn2pJ-002qqL-Qu Message is frozen
2016-09-23 02:01:22 1bmplJ-003kDV-2d Unfrozen by errmsg timer
2016-09-23 02:01:22 1bmplJ-003kDV-2d ** root@centos-67-64-minimal R=fail_remote_domains: The mail server could not deliver mail to root@centos-67-64-minimal. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.
2016-09-23 02:01:22 1bmplJ-003kDV-2d root@CentOS-67-64-minimal: error ignored
2016-09-23 02:01:22 1bnCWF-000v6a-IZ <= root@CentOS-67-64-minimal U=root P=local S=1179 T="Cron <root@CentOS-67-64-minimal> /usr/local/bin/proc /home/somepath/public_html/phs/reader.js par1" for root
2016-09-23 02:01:22 cwd=/var/spool/exim 4 args: /usr/sbin/exim -odi -Mc 1bnCWF-000v6a-IZ
2016-09-23 02:01:23 1bnCWF-000v6a-IZ ** root@centos-67-64-minimal R=fail_remote_domains: The mail server could not deliver mail to root@centos-67-64-minimal. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.
2016-09-23 02:01:23 cwd=/var/spool/exim 8 args: /usr/sbin/exim -odi -t -oem -oi -f <> -E1bnCWF-000v6a-IZ
2016-09-23 02:01:23 1bmplJ-003kDV-2d Completed
2016-09-23 02:01:23 1bmubJ-000dMx-CI Message is frozen
2016-09-23 02:01:23 1bn1rJ-002axp-El Message is frozen
2016-09-23 02:01:23 1bn2xJ-002rxH-FX Message is frozen
2016-09-23 02:01:23 1bmtHJ-000KHG-EN Message is frozen
2016-09-23 02:01:23 1bn5FJ-003XjQ-Lo Message is frozen
2016-09-23 02:01:23 1bmxzJ-001ZOK-C7 Message is frozen
2016-09-23 02:01:23 1bnCWZ-000vH3-0u <= <> R=1bnCWF-000v6a-IZ U=mailnull P=local S=2591 T="Mail delivery failed: returning message to sender" for root@CentOS-67-64-minimal
2016-09-23 02:01:23 1bmxAJ-001Kyw-Uj Message is frozen
2016-09-23 02:01:23 cwd=/var/spool/exim 4 args: /usr/sbin/exim -odi -Mc 1bnCWZ-000vH3-0u
2016-09-23 02:01:23 1bnCWZ-000vH3-0u ** root@centos-67-64-minimal R=fail_remote_domains: The mail server could not deliver mail to root@centos-67-64-minimal. The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.
2016-09-23 02:01:23 1bnCWZ-000vH3-0u Frozen (delivery error message)
2016-09-23 02:01:23 1bmz4J-001puE-To Message is frozen
2016-09-23 02:01:23 1bmsAJ-0003o9-S8 Message is frozen
2016-09-23 02:01:23 1bn9bJ-000F6h-Ip Message is frozen
2016-09-23 02:01:23 1bn6DJ-003ncT-Io Message is frozen
it seems like my server keeps sending email to itself !!
