ubuntu only use 8 cpu for RX and TX queue with vmxnet3

Question

I installed Ubuntu 16.04 on vmware esxi 6.7 with 32 core cpu for this machine. Linux only run 8 queue for tx and rx and it cant use all cpus in heavy load.

# ethtool -S ens192 | grep Queue
 Tx Queue#: 0
 Tx Queue#: 1
 Tx Queue#: 2
 Tx Queue#: 3
 Tx Queue#: 4
 Tx Queue#: 5
 Tx Queue#: 6
 Tx Queue#: 7
 Rx Queue#: 0
 Rx Queue#: 1
 Rx Queue#: 2
 Rx Queue#: 3
 Rx Queue#: 4
 Rx Queue#: 5
 Rx Queue#: 6
 Rx Queue#: 7

I config modprob

options vmxnet3 num_rqs=32 num_tqs=32

But Kernel reject options in boot

# dmesg  | grep num_
[    2.626483] vmxnet3: unknown parameter 'num_rqs' ignored
[    2.627238] vmxnet3: unknown parameter 'num_tqs' ignored

This is lspci output

lspci -k
03:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
    DeviceName: Ethernet0
    Subsystem: VMware VMXNET3 Ethernet Controller
    Kernel driver in use: vmxnet3
    Kernel modules: vmxnet3
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
    DeviceName: Ethernet1
    Subsystem: VMware VMXNET3 Ethernet Controller
    Kernel driver in use: vmxnet3
    Kernel modules: vmxnet3

and driver

# lspci -n | grep "03:00\|0b:00.0"
03:00.0 0200: 15ad:07b0 (rev 01)
0b:00.0 0200: 15ad:07b0 (rev 01)

how can i increase these queues?

Slightly off-topic, but using that many vCPUs on ESXi (and VM's in general) result is great performance loss due to CPU_WAIT. — Orphans, Oct 02 '20 at 13:24
Is your kernel up to date? The current kernel for 16.04 LTS is 4.4.0-190-generic. — Michael Hampton, Oct 02 '20 at 15:57

Anton Danilov · Accepted Answer · 2020-10-02T21:18:13.990

7

You can change the number of queues at runtime. So, check the limit of queues with ethtool --show-channels ens192, and then change the number with ethtool --set-channels ens192 rx N tx N. Note: the way above isn't applicable to the vmxnet3 driver due by lack of this feature support.

UPDATE: after inspecting of source code of the vmxnet3_driver I've found, that maximum queue count is 8 (https://elixir.bootlin.com/linux/latest/source/drivers/net/vmxnet3/vmxnet3_int.h#L306). The number of queues depends on the cpu count (https://elixir.bootlin.com/linux/latest/source/drivers/net/vmxnet3/vmxnet3_drv.c#L3398). But it cannot exceed the hardcoded value, obviously.

UPDATE2: I've checked the source of vmware-tools. The driver from it has other limits: up to 32 queues (https://github.com/vmware/open-vm-tools/blob/master/open-vm-tools/modules/shared/vmxnet/vmxnet3_defs.h#L636). So you can try to use it.

edited Oct 02 '20 at 21:18

answered Oct 02 '20 at 13:43

Anton Danilov

5,082
2
13
23

# ethtool --show-channels ens192 Channel parameters for ens192: Cannot get device channel parameters : Operation not supported – Behrad Eslamifar Oct 02 '20 at 13:48
operation not support – Behrad Eslamifar Oct 02 '20 at 13:49
i tested open-vm-tools and vmware-install.pl , both of them limited to 8 and dont detect my real cpu number – Behrad Eslamifar Oct 02 '20 at 14:33
1

Unfortunately, you don't have ability to use more queues than hardcoded in the driver source. Obviously, you can change this limit and rebuild the module, but it can cause other issues. – Anton Danilov Oct 02 '20 at 14:37
The hardcoded limit would probably correspond to the number of IRQs, which would place a hard limit at 32. – Simon Richter Oct 02 '20 at 23:44

score 1 · Answer 2 · answered Oct 07 '20 at 07:45

1

I asked this problem in open-vm-tools mailinglist

you'll need newer ESXi version (7.0), and virtual machine using hardware version 17 to have vmxnet3 device that supports more than 8 queues.

Then you'll need newer guest driver to be able to use the feature - currently driver in the Linux kernel does not support extended queues.

answered Oct 07 '20 at 07:45

Behrad Eslamifar

83
8

didnt work with this option too ;) – Behrad Eslamifar Oct 11 '20 at 08:09

ubuntu only use 8 cpu for RX and TX queue with vmxnet3

2 Answers2