In the screenshot below, the overall CPU is being reported as 3% but the mysqld process is reported as using 57%. Is this 57% of the overall 3% and thus mysqld is only using about 1.5% of the CPU?
top screenshot http://img.skitch.com/20090620-cih33piwnnrke6aw41y9p1phwr.jpg
UPDATE: A commenter below requested I press the '1' key and post the results: the 1 key http://img.skitch.com/20090620-gayja43s44qtw2wkw4aq642e8c.jpg
The 57.6% means that mysqld is using .576 of one cpu. The discrepancy is likely to be a race condition between data collection for the system as a whole and collecting the per-process data.
EDIT: Based on your update looks like you have 16 cores.
57.5% => .575/16 = .036 = 3.6%.
So that's where your 3% is coming from.
If you sum all the idle percentages and subtract from 1600%, that also comes out to about 57.5%.
Can you try pressing the number 1 when top is running and see what happens.
Edit:
Mr Unknown says it well.
top is saying "If you had one CPU core, it would be 57.5% busy."
But you have 16 cores. So this 57 number is spread out all over the place, with mysqld being multi-threaded and everything...
Kind of confusing. But I think this is because top would find it harder to display information for each core... imagine a top info line with 16 or more cores!
Also if you add up all the "%us" CPU time
1.7+1.0+2.3+1.0+7.4+11.1+15.1+8.7 = 48.3
48.3/16 = 3.01875
This is where the 3% number comes from...
us: userspace - what you're running above the kernel sy: system calls - what is running inside the kernel ni: reniced processes id: idle wa: waiting for i/o hi: hardware interrupts - how much time is spent dealing with hardware si: software interrupts - how much time is spent dealing with software-created interrupts (system calls, etc.)
I can't tell if your top is merging any multiple CPUs that you have, but that still wouldn't make a great deal of sense.
What might be happening is that top doesn't get the values for every process and the total CPU usage at the same time. There's a small amount of time where mysql may have stopped or started using a lot of CPU between the time the process info is read and the global is read. It could well just be a simple race condition.
Of course, this would have had to have happened long enough for you to have noticed, so I'm guessing this could just be a weird bug somewhere in top or the kernel's accounting.
