How can I distribute load between a NIC rx queues?

Question

I am using a Gigabit quad-port intel i350 NIC with igb driver. The NIC has been installed on a Dell machine with 32 cpu cores and it runs Ubuntu 18.04. Name of the interface I am working with is "eno2"

The interface (i.e. eno2) has 8 rx and 8 tx queues. Here is the result for ethool -l eno2:

Channel parameters for eno2:
   Pre-set maximums:
      RX:       0
      TX:       0
      Other:        1
      Combined: 8
   Current hardware settings:
      RX:       0
      TX:       0
      Other:        1
      Combined: 8

and here is the result for ls /sys/class/net/eno2/queues/ :

drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-0
drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-1
drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-2
drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-3
drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-4
drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-5
drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-6
drwxr-xr-x 2 root root 0 Sep 25 19:09 rx-7
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-0
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-1
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-2
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-3
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-4
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-5
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-6
drwxr-xr-x 3 root root 0 Sep 25 19:09 tx-7

I used ethtool -X eno2 weight 1 1 1 1 1 1 1 1 to give equal weight to all of them.

When I forward traffic to eno2, just one of its rx queues works at a time. Here is the result of ethool -X eno2 | grep rx_queue_'[0-7]'*_packets at time, let say t1:

 rx_queue_0_packets: 0
 rx_queue_1_packets: 1
 rx_queue_2_packets: 40251934
 rx_queue_3_packets: 0
 rx_queue_4_packets: 6
 rx_queue_5_packets: 3
 rx_queue_6_packets: 0
 rx_queue_7_packets: 1

if I do another experiment (let say at time t2) and run the same command, the result will be like :

 rx_queue_0_packets: 0
 rx_queue_1_packets: 1
 rx_queue_2_packets: 40251934
 rx_queue_3_packets: 0
 rx_queue_4_packets: 6
 rx_queue_5_packets: 76334041
 rx_queue_6_packets: 0
 rx_queue_7_packets: 1

As you see, just one rx queue is active each time. I would like to know how I can distribute the load between all of the rx queues?

By the way, I also tried echo ffffffff > /sys/class/net/neo2/queues/rx-0/rps_cpus to distribute load on all 32 cpu cores (using all rx queues) but it did not work.

I also tried manipulating number of rx and tx using ethrtool and it did not work either.

Any help will be highly appreciated.

How can I distribute load between a NIC rx queues?

0 Answers0