2

Scene: 16-core VMware ESX 4.1 host. Presume that the host is not over loaded in any single resource at all. Storage latencies are very low, total memory utilization is below the physical ram amount, and not a single VM exhibits any %RDY time or significant pagefile activity.

A single-processor Windows 2008 R2 VM, fully up to date with recent VMware Tools, running PeopleSoft application tier (forgive my vagueness), WebLogic, Jrockit, etc. esxtop and vcenter show physical CPU use below 70%, physical memory below 80%, the guest machine is not exhibiting signs of significant disk pagefile activity.

Users report that the machine is not processing as many events as it "should", and when they try logging in, there is significant mouse and keyboard event lag through RDP. Everything about it just feels "slow."

A second CPU is added to the VM. %RDY time does not go up appreciably, so this is judged as a sane action. Perceived and measured application performance improves dramatically.

Everyone scratches their heads, shrugs, and are happy that its fixed. Admins who have read the VMware performance troubleshooting guide are perplexed.

What did we miss?

Yolo Perdiem
  • 626
  • 1
  • 6
  • 14

1 Answers1

1

CPU Ready shows how busy the host CPUs are rather than the guest. High %RDY would probably correspond to high CPU utilization on the VM but not necessarily and if you have a lot of VMs it might not be the one in question with the issue.

From the VMWARE communities:

For every cycle of time the server has a determined number of available CPUS (including cores) to provide out to VMS. Lets say you have 2 quad core cpus. THen you have 8 available CPUS. So if you have 8 VMs running and each of them only has one vcpu then all of them get to run during that cycle. Problem occurs when you have VMs demanding more CPUs than you have available. For example if you have 6 VMS with 1 vcpus. each and 1 VM with 4 vcpus. Obviously they can't run all at the same time. VMware has to try to accomodate all the vCPUS of the 1 VM at the same time.. which means that for that one cycle either 2 VMS are on standy or the multi vCPU VM is on standby.

Certain applications that are multi-threaded will use multiple vCPUs, which will help with performance.

In your case, were you actually doing perfmon stats/logging over time to get a baseline and then doing one after the vCPU was added? That would probably give you a better indication then using the built in VMWare tools of where the differences really lie, especially at the application/process level.

TheCleaner
  • 32,627
  • 26
  • 132
  • 191
  • None of the metrics we were familiar with showed CPU-bound resources. Are there cases of applications where even if physical CPU does not reach 100% utilization, a second CPU will help? – Yolo Perdiem Dec 20 '12 at 17:17
  • I don't know for certain on your apps. That's where doing analysis on the app itself is important. Since adding the 2nd vCPU helped, it could have been contention for the physical proc, it could have been that the app allows for multi-threaded processing, etc. Using perfmon would have helped isolate this, along with process explorer. – TheCleaner Dec 20 '12 at 17:20
  • @evilensky Did you do some benchmarking or resource monitoring WITHIN the VM? – John Siu Dec 20 '12 at 22:07
  • @JohnSiu The users reported high CPU use within the guest, sure, but esxtop only showed 60-80% physical utilization. – Yolo Perdiem Jan 08 '13 at 14:51
  • @evilensky High CPU utilization maybe due to I/O delay. Check IO performance, especially on IO wait time. I believe you have to use performance monitor or 3rd party tools to investigate that, task manager don't have that information. You can also try to monitor that on the host level and see if host IO, especially storage, is max. out. – John Siu Jan 08 '13 at 15:06
  • @evilensky Since you mention adding CPU improve overall performance, maybe you want to check out my answer on another post about KVM [CPU vs IO](http://serverfault.com/a/460558/138643). You maybe in a similar situation. – John Siu Jan 08 '13 at 15:11
  • @JohnSiu IO was not maxed out, the FlashCache-assisted NetApp reported latencies < 3 ms. This was also similar to the guest's IO latencies in vcenter operations. – Yolo Perdiem Jan 08 '13 at 15:12
  • @evilensky I re-read the question. You start with 1 cpu. Maybe application within VM are actually staving for cpu time, or even waiting on each other?! – John Siu Jan 09 '13 at 04:42