-1

I wanted to try-out multiple RX/TX queue in KVM (Guest: CentOS). I have compiled DPDK (version: 18.05.1) and inserted igb_uio driver (bound two interface to it).

I am trying client to server connection (private).

client (eth1: 10.10.10.2/24) <--> (eth1) CentOS VM (DPDK: 18.05.1) (eth2) <--> server (eth1: 10.10.10.10/24)

VM manages both interface directly in passthrough mode (macvtap - passthrough).

<interface type='direct' trustGuestRxFilters='yes'>
<source dev='ens1' mode='passthrough'/>
<model type='virtio'/>
<driver name='vhost' queues='2'/>
</interface>

When l2fwd applications started with single RX & TX queue (default change) & no-mac-updating. Client & server connectivity works perfectly.

I made some changes to try multiple RX/TX queues with l2fwd application. I could see that ARP is not getting resolved at either end. VM doesn't receive any packets afterwards.

Can someone point me to the document to use multiple RX/TX queues which can verify my changes? Does multiple RX/TX queues work in VM environment? I have seen others also complaining about it.

I am newbie in DPDK world. Any help will be useful. Thank you.

Edited (Adding more details): I am configuring ethernet device with 1 RX queue and 2 TX queues in l2fwd example.

uint16_t q = 0;
uint16_t no_of_tx_queues = 2;

// Configuring Ethernet Device
rte_eth_dev_configure(portid, 1, no_of_tx_queues, &local_port_conf);

// Configuring Rx Queue
ret = rte_eth_rx_queue_setup(portid, 0, nb_rxd, rte_eth_dev_socket_id(portid), &rxq_conf, l2fwd_pktmbuf_pool);

// Configuring 2 TX Queue
for(q = 0; q < no_of_tx_queues; q++) {
    ret = rte_eth_tx_queue_setup(portid, q, nb_txd, rte_eth_dev_socket_id(portid), &txq_conf);
}

I am reading packets from single RX queue: Queue-id: 0 (as setup earlier).

nb_rx = rte_eth_rx_burst(portid, 0, pkts_burst, MAX_PKT_BURST);

I am seeing that some packets are coming and forwarded to other interface but some are not. For ICMP (ping), I can see ARP is forwarded but ICMP echo request is not read by l2fwd.

Solution what I found: I have configured 2 RX & 2 TX queues in l2fwd. I can ICMP request is read from second RX queue (Queue-id: 1) and forwarded too. With that, client to server connectivity is working as expected.

The question here is: Even I have configured 1 RX queue & 2 TX queue. Why few packets are coming on Queue-id: 1 (which is not configured & not read by l2fwd application). It is observed in KVM (running on CentOS) environment. I have checked the same on ESXI, I can see all packets are read from single queue (Queue-id: 0) and forwarded.

Why?? Please explain. Is there any way I can turn off load balancing (of packet transmitted on two RX queues) in KVM so that I can receive all the packets on single queue?

Brijesh Valera
  • 1,085
  • 2
  • 9
  • 30
  • Here are few references which I used (but couldn't fix my problem): https://doc.dpdk.org/guides/sample_app_ug/l2_forward_real_virtual.html https://mails.dpdk.org/archives/dev/2014-July/004487.html – Brijesh Valera Aug 19 '19 at 12:59

1 Answers1

0

Here is the DPDK's Vhost multiple queues test plan with all the command line arguments used: https://doc.dpdk.org/dts/test_plans/vhost_multi_queue_qemu_test_plan.html

There are no much details in the question, so the only suggestion I have is to make sure multiple queues work first and then run l2fwd on top of that. If the guest OS does not work with multiple queues, DPDK won't fix the issue.

Andriy Berestovskyy
  • 8,059
  • 3
  • 17
  • 33
  • Hi Andriy, I have added my finding and details which talks about the problem and a workaround to it. But, I couldn't understand why was it happening. Let me know if you can add some details here. Thx. – Brijesh Valera Aug 23 '19 at 06:04
  • Did not you start the VM with 2 queues? can you also share the command line used for DPDK l2fwd too? – Vipin Varghese May 07 '20 at 02:27