1

Environment:

Hypervisor: VMware ESXi, 6.5.0, 4564106

Guest OS: Debian 9.5 | Kernel: 4.19.0-0.bpo.16-amd64

Guest resources: RAM: 40GB, CPU: 16 cores Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz, Disk: VMware virtual disk on a LUN of HP SAN storage 2040 (iSCSI interface)

The VM is quite powerful in terms of resources. But, writing on the disk is unbelievably slow even on a virtual disk recently added and "resided on other LUNs". Other VMs on the same LUNs have no such problem and disk throughput is quite high and stable. This situation does not change when all applications are stopped or even when OS is switched to the single-user mode. Raw disk write is slow as well.

A sample write command:

$ dd if=/dev/zero of=./zz  ibs=1048576 count=1000 conv=fdatasync
1000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 596.236 s, 1.8 MB/s

The write throughput on another VM on the same host and on the same LUN is more than 100 MB/s.

Raw disk write if I'm not wrong about the command:

$ dd if=/dev/zero of=/dev/sdc ibs=1048576 count=1000 conv=fdatasync
1000+0 records in
2048000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 121.548 s, 8.6 MB/s

Some kernel errors are shown on VM console as below:

enter image description here

The VM is often slow with simple commands and we have occasional strange problems with response time of our applications.

Thank you.

user2798081
  • 41
  • 1
  • 5
  • What is virtual disk type? In case it is Thin Provisioned, performance could be slow. Of course, it depends on the underlying storage you have. – Stuka May 15 '21 at 20:51
  • It's thick provisioned lazy zeroed. – user2798081 May 16 '21 at 02:54
  • have many questions about your problem the vms that are working ok, have the same Guest OS? Do you check on VM properties if a disk has an I/0 limit on a disk? The VMS that are working OK, are there in the same host that the one that is working slow? If you move the VM with vmotion to another hosts is working slowly? Are you using standard or distributed switch? Is configure MTU at 9000? – JMHerrer May 22 '21 at 18:28
  • I have many questions about your problem - the vms that are working ok, have the same Guest OS? - Do you check on VM properties if a disk has an I/0 limit on a disk? - The VMS that are working OK, are there in the same host that the one that is working slow? - If you move the VM with vmotion to another hosts is working slowly? - Are you using standard or distributed switch? Is configure MTU at 9000? – JMHerrer May 22 '21 at 03:30
  • The vms that are working ok, have all Debian 9 as their guest OS. The VM properties show unlimited I/O. Both normal and slow VMs are in the same host & LUN. I will test vmotion. The switches are standard and MTU is set on its default 1500. But, this is about network. Does it have anything to do with the disk access? – user2798081 May 23 '21 at 06:16
  • We moved the VM to a different LUN and a different host, nothing changed! – user2798081 May 24 '21 at 07:20

0 Answers0