Debugging high CPU load when adding new domain inside server's control panel

Question

When adding a new domain inside my server's control panel it takes much longer than usual since some time.

CPU shows 100% usage on one core and strace reports something like this:

poll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, -64771072) = 1 ([{fd=7, revents=POLLIN}])
recvfrom(7, "OiJsb2NhbCI7fWk6NTQ0O2E6NTp7czo1"..., 8192, MSG_DONTWAIT, NULL, NULL) = 8192
poll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, -64771072) = 1 ([{fd=7, revents=POLLIN}])
recvfrom(7, "bm5hbWUiO3M6MjI6ImJhZHppZXdpYXJ6"..., 8192, MSG_DONTWAIT, NULL, NULL) = 8192
poll([{fd=7, events=POLLIN|POLLERR|POLLHUP}], 1, -64771072) = 1 ([{fd=7, revents=POLLIN}])
recvfrom(7, "IjtzOjExOiJ3ZWJtYWlscHJvZyI7czow"..., 8192, MSG_DONTWAIT, NULL, NULL) = 8192

What might be the reason ?

My OS is CentOS 5.9 64-bit. Tried different hardware and kernels and always the same...

Answer 1

Looks like one CPU is busy receiving some enormous JSON:

λ ~/ base64 -D
OiJsb2NhbCI7fWk6NTQ0O2E6NTp7czo1
bm5hbWUiO3M6MjI6ImJhZHppZXdpYXJ6
IjtzOjExOiJ3ZWJtYWlscHJvZyI7czow
^D
:"local";}i:544;a:5:{s:5
nname";s:22:"badziewiarz
";s:11:"webmailprog";s:0%

I guess that "server's control panel" software can be rewritten to be more efficient and not to degrade performance over time.

NB! Do not copy/paste straces from production servers - there could be some sensitive information inside that JSON e.g. logins/passwords/session/API-keys/etc.