This is a mail server which is running Apache2 and Postfix as services. It's been up for 135 days and nothing has been changed to the services since then. The usual load average was about 0.xx but it spiked up to around 1.00 consistently a week ago. I've checked through all the solutions regarding this issue online that I can find and investigate thoroughly on the usual culprits such as IO queue, CPU utilization, and etc. All looks normal to me.
This server is running on top of some ESXI VMWare server as VM. Some other engineer might have performed a migration at the hypervisor level. The migrations objective was to update the host servers firmware version. After that this VM was migrated back to the upgraded host. The migration is seamless , meaning the operation of this VM is not interrupted in the process.
uname -a
Linux mail.net 4.9.0-040900rc2-generic #201610232131 SMP Mon Oct 24 01:32:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 1
On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Stepping: 2
CPU MHz: 2399.569
BogoMIPS: 4799.99
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0
top
top - 17:12:23 up 135 days, 6:37, 1 user, load average: 1.00, 1.00, 1.00
Tasks: 154 total, 1 running, 153 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 12303728 total, 10122732 used, 2180996 free, 215672 buffers
KiB Swap: 1048572 total, 0 used, 1048572 free. 1321424 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 35604 4024 2660 S 0.0 0.0 0:13.76 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.92 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 2:18.67 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 4:44.58 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:18.56 watchdog/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
15 root 20 0 0 0 0 S 0.0 0.0 0:05.61 khungtaskd
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
17 root 0 -20 0 0 0 S 0.0 0.0 0:00.24 writeback
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
19 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
20 root 39 19 0 0 0 S 0.0 0.0 0:43.27 khugepaged
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
netstat -natp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 1546/rsync
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 15493/mysqld
tcp 0 0 127.0.0.1:10031 0.0.0.0:* LISTEN 1524/perl
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1058/sshd
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 1409/master
tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 1066/xinetd
tcp 0 0 203.92.11.241:25 43.245.16.27:1304 TIME_WAIT -
tcp 0 0 203.92.11.241:25 43.245.16.27:1307 ESTABLISHED 29082/smtpd
tcp 0 0 127.0.0.1:10031 127.0.0.1:52966 TIME_WAIT -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52956 TIME_WAIT -
tcp 0 0 127.0.0.1:52984 127.0.0.1:10031 CLOSE_WAIT 29082/smtpd
tcp 0 384 203.92.11.241:22 203.92.90.66:56408 ESTABLISHED 28753/sshd: xxx [
tcp 0 0 203.92.11.241:25 43.245.16.27:1305 TIME_WAIT -
tcp 0 0 203.92.11.241:25 203.92.81.184:8509 TIME_WAIT -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52962 TIME_WAIT -
tcp 1 0 127.0.0.1:52982 127.0.0.1:10031 CLOSE_WAIT 29107/smtpd
tcp 0 0 203.92.11.241:25 43.245.17.28:52703 TIME_WAIT -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52958 TIME_WAIT -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52978 TIME_WAIT -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52988 TIME_WAIT -
tcp 0 0 203.92.11.241:47072 203.92.86.79:389 ESTABLISHED 1079/sssd_be
tcp 0 0 127.0.0.1:10031 127.0.0.1:52982 FIN_WAIT2 -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52952 TIME_WAIT -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52984 FIN_WAIT2 -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52974 TIME_WAIT -
tcp 0 0 203.92.11.241:25 43.245.17.28:61916 TIME_WAIT -
tcp 0 0 127.0.0.1:10031 127.0.0.1:52968 TIME_WAIT -
tcp6 0 0 :::873 :::* LISTEN 1546/rsync
tcp6 0 0 :::80 :::* LISTEN 1788/apache2
tcp6 0 0 :::22 :::* LISTEN 1058/sshd
tcp6 0 0 :::25 :::* LISTEN 1409/master
iostat
Linux 4.9.0-040900rc2-generic (mail1.lga.net.sg) 03/10/19 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.14 0.02 0.12 0.19 0.00 99.53
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 1.26 0.07 24.53 816686 286303390
dm-0 2.04 0.07 24.53 796673 286303376
dm-1 0.00 0.00 0.00 896 0
vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 2180892 215672 1321628 0 0 0 25 0 3 0 0 100 0 0
0 0 0 2180768 215672 1321428 0 0 0 0 64 140 0 0 100 0 0
0 0 0 2180768 215672 1321432 0 0 0 115 71 157 0 0 95 5 0
0 0 0 2180768 215672 1321432 0 0 0 12 59 125 0 0 100 0 0
0 0 0 2180768 215672 1321432 0 0 0 2 62 132 0 0 100 0 0
0 0 0 2180768 215672 1321432 0 0 0 6 62 133 0 0 100 0 0
0 0 0 2185960 215672 1321432 0 0 0 0 60 131 0 0 100 0 0
0 0 0 2179896 215672 1321432 0 0 0 46 123 267 1 0 99 0 0
0 0 0 2180392 215672 1321432 0 0 0 10 66 142 0 0 100 0 0
0 0 0 2180392 215672 1321432 0 0 0 132 71 151 0 0 93 7 0
0 0 0 2180392 215672 1321432 0 0 0 4 64 134 0 0 99 0 0
0 0 0 2180392 215672 1321432 0 0 0 13 65 133 0 0 100 0 0
0 0 0 2181548 215672 1321432 0 0 0 0 64 137 0 0 100 0 0
