1

I'm running a multi-threaded application written in C++ on a Cent-OS machine. The "top" command output is showing that the application is eating most of the CPU and it's eating as system CPU (not user CPU). Then I thought of running "perf top" command, it shows:

76.02% [kernal]     [k] _spin_lock
1.87%  libc-2.12.so [.] _init_free
1.55%  [kernal]     [k] futex_wake
1.48%  libc-2.12.so [.] _init_malloc
1.37%  libc-2.12.so [.] malloc
0.77%  [kernal]     [k] __audit_syscall_exit
0.76%  libc-2.12.so [.] __lll_lock_wait_private
0.59%  libc-2.12.so [.] free

I'm not sure why it's eating that high system CPU. The application is running fine on other Linux machines with Ubuntu OS, it looks to me that it's related to CentOS.

Application is written in C++, there were around 250 active threads at the time of high system CPU.
Applications do make lots of malloc/free calls.

Operating system: Linux-2.6.32-358.18.1.el6.x86_64-x86_64-with-centos-6.4-Final
libc version: 2.12
No of cores in the machine: 16

nik_kgp
  • 1,112
  • 1
  • 9
  • 17
  • Perhaps upgrading the kernel might help. You could compile a recent kernel from source code. – Basile Starynkevitch Oct 26 '14 at 07:16
  • is this a known issue with this kernal version? do you think it's because of malloc/free calls? – nik_kgp Oct 26 '14 at 07:41
  • I'm not familiar with CentOS but my latest kernel is a 3.17.1 – Basile Starynkevitch Oct 26 '14 at 07:44
  • I suspect your spin_lock time is doing something other than `malloc` or `free`. But what is hard to tell without understanding what your application does. Can you describe what the overall code does? – Mats Petersson Oct 26 '14 at 09:26
  • Spinlocks are an opportunistic optimization of general locks which are used at the discretion of the application (developers). The magic to figuring out why your spinlocks are eating so much CPU is to determine what the application is doing while the locks are held. The spinlocks themselves aren't being slow, but rather the application is taking longer than originally anticipated to perform whatever is in the spinlock itself. Note that placing a `malloc` or `free` within a spinlock is typically a horrible idea – Mark Nunberg Oct 26 '14 at 16:12

0 Answers0