1

Recently I saw server is slowing down rapidly, and when I checked what is going on, I saw extremely large number of blkid processes and server load is enormously high. Here is the screenshot of top command:

Screenshot of top command

What to do to stop/remove those processes?

Војин Петровић
  • 111
  • 1
  • 1
    Try `sudo killall blkid`, but you will have to find out what starts these processes. – digijay Oct 07 '20 at 15:42
  • Try `pstree -ps ` for any of them, to find out what has started the process. – Michael Hampton Oct 07 '20 at 15:58
  • I'm probably just being paranoid here, but check the `/proc//exe` symlink (where `` is the process id of the `blkid` command) if it indeed points to the `blkid` command, and to the system-wide one. If the `exe` symlink doesn't point to `blkid`, or if the `blkid` it points to dwells in some suspicious directory (`/tmp/` perhaps), then your server is probably got hacked. – Lacek Oct 07 '20 at 18:49
  • I have checked `/proc//exe` on a few of them all are `/usr/sbin/blkid` ... When I kill all blkid processes they reappear, about 20 of them. And load is still high, over 150... But after each second while watching `top` PID is always changed to all of them – Војин Петровић Oct 07 '20 at 19:10
  • when pressed C button while in `top` this is the command which is run: `blkid -c /dev/null -o export` – Војин Петровић Oct 07 '20 at 19:24
  • 1
    pstree returns: `[root@packet ~]# pstree -ps 15465 systemd(1)───ds-identify(15458)───blkid(15465) ` – Војин Петровић Oct 07 '20 at 19:25
  • `[root@packet ~]# cloud-init status --long` `status: disabled` `time: Thu, 25 Jan 2018 07:10:51 +0000` `detail: DataSourceEc2` – Војин Петровић Oct 07 '20 at 20:42
  • `journalctl -u "cloud*" -b | head -100` returns `Failed to add filter for units: No data available` and there is no cloud-init data in `/var/lib/cloud/` neither in it's instance – Војин Петровић Oct 07 '20 at 20:45
  • 1
    I found identical issues on ubuntu machine https://bugs.launchpad.net/cloud-init/+bug/1856560 and tried to restart server (after almost 3 years). Problem was gone. Still wondering what happened. – Војин Петровић Oct 07 '20 at 20:58
  • *almost 3 years*? With the PIDs are quite far apart, I wonder if the number of leftover processes coincides with the number of systemd daemon-reexecs in that timespan, which might point to a separate bug in *systemd.generator* cleanup.. – anx Oct 07 '20 at 21:43
  • Anx, I'm not expert in linux systems so I don't know what are R and D states. And server was 3 years online, and this happened few days ago. – Војин Петровић Oct 07 '20 at 22:11

0 Answers0