1

While using ps aux to find the PID of a process, I came across some output that prevented me from doing so. I've done this before, on this same server, within the last month. Today's output looked like:

(Scroll right → → → →)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1 192652  2472 ?        Ss    2017 194:50 [systemd]
root         2  0.0  0.0      0     0 ?        S     2017   0:00 [kthreadd/153968]
root         3  0.0  0.0      0     0 ?        S     2017   1:47 [khelper/153968]
root        95  0.0  0.0  41608   120 ?        Ss    2017   0:00 !!        ?\???  ?       ????
root       115  0.0  0.1 586308  2760 ?        Ssl   2017  18:36 !!        ?\???  ?
postfix    117  0.0  0.0 404692   516 ?        Ssl   2017  10:33 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @     ?       8       ?       ?       ?
dbus       131  0.0  0.0  24412   900 ?        Ss    2017 123:06 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @     ?
root       166  0.0  0.0  11696   524 ?        Ss    2017   2:36 !!        ?\???  ?
root       168  0.0  0.0  80308   280 ?        Ss    2017   0:55 !!        ?\???
root       175  0.0  0.0  27060   408 ?        Ss    2017   0:07 !!        ?\???  ?       ????    ?        ?      ?
dovecot    182  0.0  0.0   9384   428 ?        S     2017   1:19 !!        ?\?
root       183  0.0  0.0   9516   548 ?        S     2017   0:47 !!        ?
root       204  0.0  0.0 124168   440 ?        Ss    2017   7:01 !!        ?\???  ?
root       205  0.0  0.0   6404     8 tty2     Ss+   2017   0:00 !!        ?\???  ?       ????
root       206  0.0  0.0 109984     8 tty1     Ss+   2017   0:00 !!        ?\???  ?       ????    ?        ?      ?       d       ?
named      239  0.0  0.0 243560  2036 ?        Ssl   2017   0:06 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @
mysql      264  0.0  0.0 113208     8 ?        Ss    2017   0:00 !!        ?\???  ?       ????    ?        ?
root       582  0.0  1.3 823592 28404 ?        Sl    2017 327:08 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @     ?       8       ?       ?       ?
root       799  0.0  0.0  12064   580 ?        S     2017  19:14 !!        ?\???  ?
root       998  0.0  0.0  88896   456 ?        Ss    2017  45:47 !!        ?\???  ?       ????
postfix   1045  0.0  0.0  89196   720 ?        S     2017  34:05 !!        ?\???  ?
postfix   1083  0.0  0.0  89104   520 ?        S     2017   1:33 !!        ?\???  ?
mysql     1088  0.1 14.2 3096344 299164 ?      Sl    2017 934:25 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @     ?       8       ?       ?       ?        ???H?  ?               ?       ??@     ?       ??      ?       ??      ?       ??      ?       ??      ?
drweb     1238  0.0  7.2 394364 153036 ?       Ss    2017   5:19 !!        ?\???  ?       ????    ?
root      2696  0.0  0.0  24220  1692 ?        Ss   16:41   0:01 !!        ?\???  ?       ????
root      2819  0.0  0.3  36768  6440 ?        Ss   16:42   0:00 /usr/lib/systemd/systemd-journald
root      4231  0.0  0.2 272824  4972 ?        Ss   01:09   0:15 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @     ?       8       ?       ?       ?        ???H?  ?               ?       ??@     ?       ??      ?       ??      ?       ??      ?       ??      ?
popuser   4232  0.0  0.1 272824  2200 ?        S    01:09   0:00 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @     ?       8       ?       ?       ?        ???H?  ?               ?       ??@     ?       ??      ?       ??      ?       ??      ?       ??      ?
drweb     4296  0.0  7.2 394364 151984 ?       S    01:09   0:01 !!        ?\???  ?       ????    ?
root      6223  0.0  0.1 507440  2348 ?        Ss   May13  12:06 !!        ?\???  ?       ????    ?        ?
root      6626  0.0  0.1 472820  3784 ?        Ss   May13   1:05 !!        ?\???  ?       ????    ?        ?      ?       d       ?       @ @     ?       8       ?       ?       ?        ???H?  ?               ?
root      7405  0.0  0.2 267892  5980 ?        Ss   May11   6:44 !!        ?\???  ?       ???
root      8205  0.0  0.0  52784  1656 ?        Ss   May08   0:00 !!        ?\???  ?       ????    ?        ?      ?
apache    8360  0.0  0.0 266712  1832 ?        S    Jul22   0:00 !!        ?\???  ?       ???
apache    8361  0.0  0.0 267892  1740 ?        S    Jul22   0:03 !!        ?\???  ?       ???
apache    8362  0.0  0.3 2248416 7956 ?        Sl   Jul22   1:52 !!        ?\???  ?       ???
apache    8363  0.0  0.3 2248416 8204 ?        Sl   Jul22   1:31 !!        ?\???  ?       ???
apache    8389  0.0  0.4 2248416 8668 ?        Sl   Jul22   2:38 !!        ?\???  ?       ???
apache    8577  0.0  0.4 2248416 9488 ?        Sl   Jul22   3:54 !!        ?\???  ?       ???
root     12484  0.0  0.1 217092  3336 ?        Ss   Jul05   7:00 !!        ?\???  ?       ????    ?        ?      ?       d
# ... snip ...

There are some valid commands shown later in the output. However, the bulk of ps output contains this junk !! ?\??? ? output. But, as you can see many of these commands have been running for a long time—they used to be valid commands too.

First, if there's a valid reason for this output ... what is it?

Second, is this something I should worry about?

Robert K
  • 572
  • 1
  • 5
  • 12
  • What are the contents of, e.g., `/proc/6223/cmdline` and `/proc/6223/stat`? – Mark Wagner Jul 25 '18 at 23:29
  • @MarkWagner Turns out, the hosting company did a forced kernel update and restarted all affected servers. I don't know if it's related, but I won't be able to read the affected data. (The issue is gone after the kernel rollback and forced reboot.) – Robert K Jul 26 '18 at 13:23
  • Did the hosting company say why? That is a red flag that something was compromised. – Mark Wagner Jul 26 '18 at 17:06
  • @MarkWagner I don't have the exact text handy, but a paraphrase is: "We pushed a kernel update that was defective, and had to roll it back." Kicker is, they didn't inform us! Instead, we got notifications of restarts across the 10+ accounts we're on. It's the last straw, honestly. We'll be migrating away from these guys for sure (and they're a pretty big name provider, too). – Robert K Jul 27 '18 at 13:10

1 Answers1

0

I haven't seen this before, so I can only guess. But I think that the memory pages containing the arguments to the processes have been paged out to swap space, and that the pages are not read back from swap just to show the arguments.

The command ps aux reads the commandline from /proc/$PID/cmdline . You can look at a hex dump of some of the processes, as well as other processes where the output is correct. If the valid output is for newer processes, that would support this theory. Since last month when it worked you probably hat some process that consumed much memory and caused unused pages to be written to swap. The program arguments are rarely needed after a program is started, so unless there is something else on the same memory page, there is no reason to load those pages into memory again.

RalfFriedl
  • 3,108
  • 4
  • 13
  • 17