CentOS7 kswapd0 process eating 99% CPU

Question

CentOS7 root eating 99% CPU

When I run top on centos7, I found kswapd0 eating 99% CPU, as shown in the screenshot above.

Below is some other information

top - 11:38:04 up 32 days, 16:14,  1 user,  load average: 0.06, 0.13, 0.19
Tasks: 142 total,   2 running, 140 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.0 us,  2.0 sy,  0.0 ni, 79.0 id,  6.7 wa,  0.0 hi,  0.0 si,  0.3 st
KiB Mem :  3618860 total,   199684 free,  1130200 used,  2288976 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2152124 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
12493 mysql     20   0 1200596 402140   6868 S  5.3 11.1  46:54.22 mysqld
13460 apache    20   0 1089628  43932  22264 S  3.0  1.2   1:16.40 php-fpm
13462 apache    20   0 1088772  42704  21672 S  1.3  1.2   1:14.47 php-fpm
 9391 root      20   0  124508  11588   4852 S  0.7  0.3   8:38.31 AliYunDun
13474 apache    20   0 1073892  40976  19884 S  0.7  1.1   1:15.93 php-fpm
12609 apache    20   0 1088832  44976  24000 S  0.3  1.2   1:15.94 php-fpm
12615 apache    20   0 1077948  42776  21764 S  0.3  1.2   1:16.34 php-fpm
12616 apache    20   0 1077568  43116  22344 S  0.3  1.2   1:14.59 php-fpm
12619 apache    20   0 1085672  44144  22380 S  0.3  1.2   1:14.62 php-fpm
12663 nginx     20   0   66928   7604   2276 S  0.3  0.2   4:55.48 nginx
12664 nginx     20   0   68064   9124   2708 S  0.3  0.3   5:17.61 nginx
13451 apache    20   0 1077360  41092  20452 S  0.3  1.1   1:15.16 php-fpm
20987 root      20   0  143572   5788   4492 S  0.3  0.2   0:00.75 sshd
27671 root      20   0  157676   2208   1544 R  0.3  0.1   0:00.12 top
28370 apache    20   0 1071564  34900  15676 S  0.3  1.0   1:01.53 php-fpm
    1 root      20   0  191276   5412   1464 S  0.0  0.1   2:03.59 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.06 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   0:46.51 ksoftirqd/0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
    8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/0
   10 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/1
   11 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/2
   12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/3
   13 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/4
   14 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/5
   15 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/6
   16 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/7
   17 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/8
   18 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/9
   19 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/10
   20 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/11
   21 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/12
   22 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/13
   23 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcuob/14

I dont know why

[root@fl basic]#free -lh
              total        used        free      shared  buff/cache   available
Mem:           3.5G        1.1G        197M         87M        2.2G        2.1G
Low:           3.5G        3.3G        197M
High:            0B          0B          0B
Swap:            0B          0B          0B

score 3 · Accepted Answer · answered Jun 06 '17 at 06:10

3

Well, root don't eat your CPU, it do kswapd. There is known bug in linux kernel. We have same issues in CentOS 7.1 after upgrading to 7.2 it has gone. May be kernel update would be solution for you.

answered Jun 06 '17 at 06:10

Alexander Tolkachev

4,608
3
14
23

shell run `rpm --query centos-release` output `centos-release-7-2.1511.el7.centos.2.10.x86_64` – Mario7 Jun 07 '17 at 06:19
@Mario7 what kernel version? – Alexander Tolkachev Jun 07 '17 at 06:32
uname -r `3.10.0-327.36.3.el7.x86_64` – Mario7 Jun 11 '17 at 12:00
@Mario7 we don't have problems with same kernel. – Alexander Tolkachev Jun 11 '17 at 12:34

score 2 · Answer 2 · answered Jun 06 '17 at 06:55

2

As stated by AlexanderT, it can be a kernel bug and so you should update your kernel.

If you can't do that, a possible workaround is to periodically drop the buffer/pagecache by issuing the following command: echo 3 > /proc/sys/vm/drop_caches

answered Jun 06 '17 at 06:55

shodanshok

47,711
7
111
180

score 0 · Answer 3 · answered May 27 '20 at 23:51

While in this case it looks like the kernel bug mentioned above, for other cases it may be something else.

It could be malware. The process name has been changed to display kswapd0. This is a common trick by malware, and kswapd0 is one I've specifically seen.

Some other clues this isn't just a benign kernel process:

kernel processes doesn't have surrounding [brackets] around the name
kernel processes are usually lower PIDs; kswapd are around 500 on a server with 64 cores (and more cores mean more kprocs like migration, watchdog, ksoftirqd, which run before kswapd)
files in /proc/$PID/fd - a true kproc won't have any file descriptors.
VIRT/RES/SHR will be 0 for kernel process, but a fake one will have non-zero values here.
look for other suspicious processes. Do you see things like uptime, ls -la, id, whoami, etc that normally return in a fraction of a second? How are they all running the instant you ran ps? You might have malware.

You could still possibly have a genuine swap load. If your system was low on free memory, or your swap had activity (if you have one), then it may be due to normal VM subsystem operation.

CentOS7 kswapd0 process eating 99% CPU

3 Answers3