cat /proc/pid/cmdline hangs (no cgroups, no nfs or network fs, no memory issues)

Question

We are a bit stumped. Once in a while ps/w hang forever (at least an hour) on our server, and it is on reading /proc/pid/cmdline for a process (happend twice today, verified with strace cat directly on cmdline). Only things that fixes it is a kill -9.

There are no containers running on the server, and no OOM conditions (actually >3TB memory free). No NFS or other network filesystems.

Running 5.19.0-1019-aws #20~22.04.1-Ubuntu SMP Thu Jan 26 11:24:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux, Ubuntu 22, its a high core-count and high memory NUMA system.

The offending processes are at S or Z states during the hang.

We read the following but it didn't help us:

• https://rachelbythebay.com/w/2014/10/27/ps/
• https://news.ycombinator.com/item?id=8520045

ChatGPT4 is stumped to.

Any ideas? Thanks!