The issue
I have a server that reaches pm.max_children (200) without spike in CPU usage. No network traffic (MB out / MB in) but extremely high number of socket allocated that do not get closed.
CPU: 16 Cores RAM: 64 GB
Pictures from my Prometheus monitor system.
Network Traffic, Look at the hole
No sockets get to TCP_tw point, They get stuck on TCP_alloc
Huge spike in socket use
To mitigate this problem i have to manually restart php-fpm. Otherwise its completely stuck for new connections.
CPU usage, It drops hard
Update
I suspect this is caused by tables getting locked in the database for an extended period of time which causes all connections trying to reach that data to wait and it ends up getting stuck in a big pile of connections.
I am currently setting up a Mysql exporter to see if i can find more.
