0

I am running Wind River Linux LTS and I am trying to add support to BNXT_EN series NICs. The issue I am facing is that the machine is not detecting nics when the dpdk application is started. Kindly help me troubleshoot this issue. I did google a lot about this. I thought NICs should bound to userspace for them to work (igb_uio) tried to bind manually using dpdk-devbind.py, but the NIC disappears when I do that. Binding it back to bnxt_en, makes it come up, but not work. Thanks in advance. Following are some details that might help:

Edit-1

  • As you can see from below, librte_pmd_bnxt.so is built and loaded
  • I did set the log level to 8 and dpdk logs are added below.
bash# find . -name "librte*bnxt*"
./usr/lib64/dpdk-18/librte_pmd_bnxt.so.2.1
bash#
bash# cat /sys/bus/pci/drivers/bnxt_en/
0000:00:03.0/ 0000:00:06.0/ module/       remove_id     unbind
0000:00:05.0/ bind          new_id        uevent

After binding them to igb_uio manually:

bash# cat /sys/bus/pci/drivers/igb_uio/
0000:00:03.0/ 0000:00:06.0/ module/       remove_id     unbind
0000:00:05.0/ bind          new_id        uevent
bash# cat /mnt/disk0/dpdk.log
03-02-2021  10:49:43.406347 EAL: Support maximum 128 logical core(s) by configuration.
03-02-2021  10:49:43.406352 EAL: Detected 16 lcore(s)
03-02-2021  10:49:43.406365 EAL: Detected 1 NUMA nodes
03-02-2021  10:49:43.406664 EAL: open shared lib /usr/lib64/dpdk-18/librte_pmd_ixgbe.so.2.1
03-02-2021  10:49:43.407327 EAL: open shared lib /usr/lib64/dpdk-18/librte_pmd_e1000.so.1.1
03-02-2021  10:49:43.409000 EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
03-02-2021  10:49:43.409957 EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
03-02-2021  10:49:43.409963 EAL: VFIO PCI modules not loaded
03-02-2021  10:49:43.411620 EAL: No free hugepages reported in hugepages-1048576kB
03-02-2021  10:49:43.411767 EAL: Probing VFIO support...
03-02-2021  10:49:43.411777 EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
03-02-2021  10:49:43.411780 EAL: VFIO modules not loaded, skipping VFIO support...
03-02-2021  10:49:43.411792 EAL: Ask a virtual area of 0x2e000 bytes
03-02-2021  10:49:43.411813 EAL: Virtual area found at 0x100000000 (size = 0x2e000)
03-02-2021  10:49:43.411954 EAL: Setting up physically contiguous memory...
03-02-2021  10:49:43.411964 EAL: Setting maximum number of open files to 4096
03-02-2021  10:49:43.411970 EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
03-02-2021  10:49:43.411972 EAL: Detected memory type: socket_id:0 hugepage_sz:2097152
03-02-2021  10:49:43.411976 EAL: Creating 4 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
03-02-2021  10:49:43.411979 EAL: Ask a virtual area of 0x1000 bytes
03-02-2021  10:49:43.411983 EAL: Virtual area found at 0x10002e000 (size = 0x1000)
03-02-2021  10:49:43.411999 EAL: Memseg list allocated: 0x100000kB at socket 0
03-02-2021  10:49:43.412002 EAL: Ask a virtual area of 0x800000000 bytes
03-02-2021  10:49:45.37931 EAL: Virtual area found at 0x140000000 (size = 0x800000000)
03-02-2021  10:49:45.52277 EAL: Memseg list allocated: 0x100000kB at socket 0
03-02-2021  10:49:45.52281 EAL: Ask a virtual area of 0x800000000 bytes
03-02-2021  10:49:46.680676 EAL: Virtual area found at 0x980000000 (size = 0x800000000)
03-02-2021  10:49:46.694952 EAL: Ask a virtual area of 0x1000 bytes
03-02-2021  10:49:46.694970 EAL: Virtual area found at 0x1180000000 (size = 0x1000)
03-02-2021  10:49:46.695028 EAL: Memseg list allocated: 0x100000kB at socket 0
03-02-2021  10:49:46.695031 EAL: Ask a virtual area of 0x800000000 bytes
03-02-2021  10:49:48.305421 EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
03-02-2021  10:49:48.319722 EAL: Ask a virtual area of 0x1000 bytes
03-02-2021  10:49:48.319741 EAL: Virtual area found at 0x19c0000000 (size = 0x1000)
03-02-2021  10:49:48.319798 EAL: Memseg list allocated: 0x100000kB at socket 0
03-02-2021  10:49:48.319827 EAL: Ask a virtual area of 0x800000000 bytes
03-02-2021  10:49:49.950396 EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
03-02-2021  10:49:49.964673 EAL: Creating 32 segment lists: n_segs:1024 socket_id:0 hugepage_sz:2097152
03-02-2021  10:49:49.964684 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:49.964702 EAL: Virtual area found at 0x2200000000 (size = 0xd000)
03-02-2021  10:49:49.964809 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:49.964813 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.62627 EAL: Virtual area found at 0x2200200000 (size = 0x80000000)
03-02-2021  10:49:50.62681 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.62690 EAL: Virtual area found at 0x2280200000 (size = 0xd000)
03-02-2021  10:49:50.62782 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.62785 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.161349 EAL: Virtual area found at 0x2280400000 (size = 0x80000000)
03-02-2021  10:49:50.161395 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.161404 EAL: Virtual area found at 0x2300400000 (size = 0xd000)
03-02-2021  10:49:50.161493 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.161496 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.259251 EAL: Virtual area found at 0x2300600000 (size = 0x80000000)
03-02-2021  10:49:50.259294 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.259302 EAL: Virtual area found at 0x2380600000 (size = 0xd000)
03-02-2021  10:49:50.259393 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.259397 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.358313 EAL: Virtual area found at 0x2380800000 (size = 0x80000000)
03-02-2021  10:49:50.358354 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.358363 EAL: Virtual area found at 0x2400800000 (size = 0xd000)
03-02-2021  10:49:50.358447 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.358453 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.456097 EAL: Virtual area found at 0x2400a00000 (size = 0x80000000)
03-02-2021  10:49:50.456138 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.456146 EAL: Virtual area found at 0x2480a00000 (size = 0xd000)
03-02-2021  10:49:50.456227 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.456230 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.555212 EAL: Virtual area found at 0x2480c00000 (size = 0x80000000)
03-02-2021  10:49:50.555256 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.555266 EAL: Virtual area found at 0x2500c00000 (size = 0xd000)
03-02-2021  10:49:50.555341 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.555344 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.653819 EAL: Virtual area found at 0x2500e00000 (size = 0x80000000)
03-02-2021  10:49:50.653864 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.653874 EAL: Virtual area found at 0x2580e00000 (size = 0xd000)
03-02-2021  10:49:50.653967 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.653970 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.752922 EAL: Virtual area found at 0x2581000000 (size = 0x80000000)
03-02-2021  10:49:50.752966 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.752975 EAL: Virtual area found at 0x2601000000 (size = 0xd000)
03-02-2021  10:49:50.753065 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.753070 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.850820 EAL: Virtual area found at 0x2601200000 (size = 0x80000000)
03-02-2021  10:49:50.850861 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.850870 EAL: Virtual area found at 0x2681200000 (size = 0xd000)
03-02-2021  10:49:50.850939 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.850942 EAL: A[ 7668.824713] serial8250: too much work for irq4
[ 7668.826866] new_iir c1 ier 7 fcr c2 lcr 13 mcr b lsr 60 msr b0 rx 0 dll 0 dlh 7
sk a virtual area of 0x80000000 bytes
03-02-2021  10:49:50.949796 EAL: Virtual area found at 0x2681400000 (size = 0x80000000)
03-02-2021  10:49:50.949850 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:50.949860 EAL: Virtual area found at 0x2701400000 (size = 0xd000)
03-02-2021  10:49:50.949929 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:50.949932 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.47666 EAL: Virtual area found at 0x2701600000 (size = 0x80000000)
03-02-2021  10:49:51.47707 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.47716 EAL: Virtual area found at 0x2781600000 (size = 0xd000)
03-02-2021  10:49:51.47784 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.47788 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.146429 EAL: Virtual area found at 0x2781800000 (size = 0x80000000)
03-02-2021  10:49:51.146469 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.146477 EAL: Virtual area found at 0x2801800000 (size = 0xd000)
03-02-2021  10:49:51.146546 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.146551 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.244248 EAL: Virtual area found at 0x2801a00000 (size = 0x80000000)
03-02-2021  10:49:51.244287 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.244295 EAL: Virtual area found at 0x2881a00000 (size = 0xd000)
03-02-2021  10:49:51.244357 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.244360 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.343355 EAL: Virtual area found at 0x2881c00000 (size = 0x80000000)
03-02-2021  10:49:51.343396 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.343404 EAL: Virtual area found at 0x2901c00000 (size = 0xd000)
03-02-2021  10:49:51.343468 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.343472 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.441217 EAL: Virtual area found at 0x2901e00000 (size = 0x80000000)
03-02-2021  10:49:51.441255 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.441265 EAL: Virtual area found at 0x2981e00000 (size = 0xd000)
03-02-2021  10:49:51.441331 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.441334 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.540519 EAL: Virtual area found at 0x2982000000 (size = 0x80000000)
03-02-2021  10:49:51.540558 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.540566 EAL: Virtual area found at 0x2a02000000 (size = 0xd000)
03-02-2021  10:49:51.540635 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.540640 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.639056 EAL: Virtual area found at 0x2a02200000 (size = 0x80000000)
03-02-2021  10:49:51.639096 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.639106 EAL: Virtual area found at 0x2a82200000 (size = 0xd000)
03-02-2021  10:49:51.639169 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.639172 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.738309 EAL: Virtual area found at 0x2a82400000 (size = 0x80000000)
03-02-2021  10:49:51.738354 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.738363 EAL: Virtual area found at 0x2b02400000 (size = 0xd000)
03-02-2021  10:49:51.738427 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.738430 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.836237 EAL: Virtual area found at 0x2b02600000 (size = 0x80000000)
03-02-2021  10:49:51.836276 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.836286 EAL: Virtual area found at 0x2b82600000 (size = 0xd000)
03-02-2021  10:49:51.836351 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.836354 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:51.935570 EAL: Virtual area found at 0x2b82800000 (size = 0x80000000)
03-02-2021  10:49:51.935621 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:51.935631 EAL: Virtual area found at 0x2c02800000 (size = 0xd000)
03-02-2021  10:49:51.935706 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:51.935720 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.33555 EAL: Virtual area found at 0x2c02a00000 (size = 0x80000000)
03-02-2021  10:49:52.33609 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.33618 EAL: Virtual area found at 0x2c82a00000 (size = 0xd000)
03-02-2021  10:49:52.33685 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.33688 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.132378 EAL: Virtual area found at 0x2c82c00000 (size = 0x80000000)
03-02-2021  10:49:52.132420 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.132429 EAL: Virtual area found at 0x2d02c00000 (size = 0xd000)
03-02-2021  10:49:52.132502 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.132505 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.230175 EAL: Virtual area found at 0x2d02e00000 (size = 0x80000000)
03-02-2021  10:49:52.230216 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.230227 EAL: Virtual area found at 0x2d82e00000 (size = 0xd000)
03-02-2021  10:49:52.230295 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.230298 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.328849 EAL: Virtual area found at 0x2d83000000 (size = 0x80000000)
03-02-2021  10:49:52.328887 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.328895 EAL: Virtual area found at 0x2e03000000 (size = 0xd000)
03-02-2021  10:49:52.328958 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.328963 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.426649 EAL: Virtual area found at 0x2e03200000 (size = 0x80000000)
03-02-2021  10:49:52.426687 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.426695 EAL: Virtual area found at 0x2e83200000 (size = 0xd000)
03-02-2021  10:49:52.426756 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.426759 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.525606 EAL: Virtual area found at 0x2e83400000 (size = 0x80000000)
03-02-2021  10:49:52.525645 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.525653 EAL: Virtual area found at 0x2f03400000 (size = 0xd000)
03-02-2021  10:49:52.525718 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.525721 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.623328 EAL: Virtual area found at 0x2f03600000 (size = 0x80000000)
03-02-2021  10:49:52.623368 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.623380 EAL: Virtual area found at 0x2f83600000 (size = 0xd000)
03-02-2021  10:49:52.623447 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.623450 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.722537 EAL: Virtual area found at 0x2f83800000 (size = 0x80000000)
03-02-2021  10:49:52.722589 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.722598 EAL: Virtual area found at 0x3003800000 (size = 0xd000)
03-02-2021  10:49:52.722669 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.722672 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.821293 EAL: Virtual area found at 0x3003a00000 (size = 0x80000000)
03-02-2021  10:49:52.821339 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.821348 EAL: Virtual area found at 0x3083a00000 (size = 0xd000)
03-02-2021  10:49:52.821423 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:52.821426 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:52.920409 EAL: Virtual area found at 0x3083c00000 (size = 0x80000000)
03-02-2021  10:49:52.920452 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:52.920461 EAL: Virtual area found at 0x3103c00000 (size = 0xd000)
03-02-2021  10:49:52.920535 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:[ 7668.970809] serial8250: too much work for irq4
[ 7668.972922] new_iir c1 ier 7 fcr c2 lcr 13 mcr b lsr 60 msr b0 rx 0 dll 0 dlh 7
49:52.920538 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:53.18368 EAL: Virtual area found at 0x3103e00000 (size = 0x80000000)
03-02-2021  10:49:53.18413 EAL: Ask a virtual area of 0xd000 bytes
03-02-2021  10:49:53.18435 EAL: Virtual area found at 0x3183e00000 (size = 0xd000)
03-02-2021  10:49:53.18511 EAL: Memseg list allocated: 0x800kB at socket 0
03-02-2021  10:49:53.18514 EAL: Ask a virtual area of 0x80000000 bytes
03-02-2021  10:49:53.117344 EAL: Virtual area found at 0x3184000000 (size = 0x80000000)
03-02-2021  10:49:53.117674 EAL: Trying to obtain current memory policy.
03-02-2021  10:49:53.403560 EAL: Hugepage /hugepages/rtemap_55 is on socket 0
03-02-2021  10:49:53.403562 EAL: Hugepage /hugepages/rtemap_54 is on socket 0
03-02-2021  10:49:53.403564 EAL: Hugepage /hugepages/rtemap_53 is on socket 0
03-02-2021  10:49:53.403567 EAL: Hugepage /hugepages/rtemap_52 is on socket 0
03-02-2021  10:49:53.403569 EAL: Hugepage /hugepages/rtemap_51 is on socket 0
03-02-2021  10:49:53.403572 EAL: Hugepage /hugepages/rtemap_50 is on socket 0
03-02-2021  10:49:53.403574 EAL: Hugepage /hugepages/rtemap_49 is on socket 0
03-02-2021  10:49:53.403576 EAL: Hugepage /hugepages/rtemap_48 is on socket 0
03-02-2021  10:49:53.403579 EAL: Hugepage /hugepages/rtemap_47 is on socket 0
03-02-2021  10:49:53.403581 EAL: Hugepage /hugepages/rtemap_46 is on socket 0
03-02-2021  10:49:53.403600 EAL: Hugepage /hugepages/rtemap_45 is on socket 0
03-02-2021  10:49:53.403603 EAL: Hugepage /hugepages/rtemap_44 is on socket 0
03-02-2021  10:49:53.403605 EAL: Hugepage /hugepages/rtemap_43 is on socket 0
03-02-2021  10:49:53.403607 EAL: Hugepage /hugepages/rtemap_42 is on socket 0
03-02-2021  10:49:53.403610 EAL: Hugepage /hugepages/rtemap_41 is on socket 0
03-02-2021  10:49:53.403612 EAL: Hugepage /hugepages/rtemap_40 is on socket 0
03-02-2021  10:49:53.403614 EAL: Hugepage /hugepages/rtemap_39 is on socket 0
03-02-2021  10:49:53.403616 EAL: Hugepage /hugepages/rtemap_38 is on socket 0
03-02-2021  10:49:53.403619 EAL: Hugepage /hugepages/rtemap_37 is on socket 0
03-02-2021  10:49:53.403621 EAL: Hugepage /hugepages/rtemap_36 is on socket 0
03-02-2021  10:49:53.403624 EAL: Hugepage /hugepages/rtemap_35 is on socket 0
03-02-2021  10:49:53.403626 EAL: Hugepage /hugepages/rtemap_34 is on socket 0
03-02-2021  10:49:53.403643 EAL: Hugepage /hugepages/rtemap_33 is on socket 0
03-02-2021  10:49:53.403645 EAL: Hugepage /hugepages/rtemap_32 is on socket 0
03-02-2021  10:49:53.403648 EAL: Hugepage /hugepages/rtemap_31 is on socket 0
03-02-2021  10:49:53.403650 EAL: Hugepage /hugepages/rtemap_30 is on socket 0
03-02-2021  10:49:53.403653 EAL: Hugepage /hugepages/rtemap_29 is on socket 0
03-02-2021  10:49:53.403655 EAL: Hugepage /hugepages/rtemap_28 is on socket 0
03-02-2021  10:49:53.403657 EAL: Hugepage /hugepages/rtemap_27 is on socket 0
03-02-2021  10:49:53.403659 EAL: Hugepage /hugepages/rtemap_26 is on socket 0
03-02-2021  10:49:53.403662 EAL: Hugepage /hugepages/rtemap_25 is on socket 0
03-02-2021  10:49:53.403665 EAL: Hugepage /hugepages/rtemap_24 is on socket 0
03-02-2021  10:49:53.403667 EAL: Hugepage /hugepages/rtemap_23 is on socket 0
03-02-2021  10:49:53.403689 EAL: Hugepage /hugepages/rtemap_22 is on socket 0
03-02-2021  10:49:53.403691 EAL: Hugepage /hugepages/rtemap_21 is on socket 0
03-02-2021  10:49:53.403694 EAL: Hugepage /hugepages/rtemap_20 is on socket 0
03-02-2021  10:49:53.403696 EAL: Hugepage /hugepages/rtemap_19 is on socket 0
03-02-2021  10:49:53.403698 EAL: Hugepage /hugepages/rtemap_18 is on socket 0
03-02-2021  10:49:53.403701 EAL: Hugepage /hugepages/rtemap_17 is on socket 0
03-02-2021  10:49:53.403704 EAL: Hugepage /hugepages/rtemap_16 is on socket 0
03-02-2021  10:49:53.403706 EAL: Hugepage /hugepages/rtemap_15 is on socket 0
03-02-2021  10:49:53.403708 EAL: Hugepage /hugepages/rtemap_14 is on socket 0
03-02-2021  10:49:53.403710 EAL: Hugepage /hugepages/rtemap_13 is on socket 0
03-02-2021  10:49:53.403712 EAL: Hugepage /hugepages/rtemap_12 is on socket 0
03-02-2021  10:49:53.403767 EAL: Hugepage /hugepages/rtemap_11 is on socket 0
03-02-2021  10:49:53.403770 EAL: Hugepage /hugepages/rtemap_10 is on socket 0
03-02-2021  10:49:53.403772 EAL: Hugepage /hugepages/rtemap_9 is on socket 0
03-02-2021  10:49:53.403774 EAL: Hugepage /hugepages/rtemap_8 is on socket 0
03-02-2021  10:49:53.403776 EAL: Hugepage /hugepages/rtemap_7 is on socket 0
03-02-2021  10:49:53.403779 EAL: Hugepage /hugepages/rtemap_6 is on socket 0
03-02-2021  10:49:53.403781 EAL: Hugepage /hugepages/rtemap_5 is on socket 0
03-02-2021  10:49:53.403784 EAL: Hugepage /hugepages/rtemap_4 is on socket 0
03-02-2021  10:49:53.403786 EAL: Hugepage /hugepages/rtemap_3 is on socket 0
03-02-2021  10:49:53.403788 EAL: Hugepage /hugepages/rtemap_2 is on socket 0
03-02-2021  10:49:53.403790 EAL: Hugepage /hugepages/rtemap_1 is on socket 0
03-02-2021  10:49:53.403793 EAL: Hugepage /hugepages/rtemap_0 is on socket 0
03-02-2021  10:49:53.404818 EAL: Requesting 59 pages of size 2MB from socket 0
03-02-2021  10:49:53.405062 EAL: Attempting to map 118M on socket 0
03-02-2021  10:49:53.407277 EAL: Allocated 118M on socket 0
03-02-2021  10:49:56.136181 EAL: Added 118M to heap on socket 0
03-02-2021  10:49:56.236341 EAL: TSC frequency is ~1995315 KHz
03-02-2021  10:49:56.236396 EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
03-02-2021  10:49:56.236513 EAL: Master lcore 0 is ready (tid=7fc631837480;cpuset=[0])
03-02-2021  10:49:56.243340 EAL: lcore 7 is ready (tid=7fc62dff0700;cpuset=[6])
03-02-2021  10:49:56.243747 EAL: lcore 6 is ready (tid=7fc62e0f1700;cpuset=[5])
03-02-2021  10:49:56.243851 EAL: lcore 5 is ready (tid=7fc62e1f2700;cpuset=[4])
03-02-2021  10:49:56.244261 EAL: lcore 8 is ready (tid=7fc62deef700;cpuset=[7])
03-02-2021  10:49:56.244362 EAL: lcore 4 is ready (tid=7fc62e2f3700;cpuset=[3])
03-02-2021  10:49:56.244746 EAL: lcore 9 is ready (tid=7fc62ddee700;cpuset=[8])
03-02-2021  10:49:56.244845 EAL: lcore 10 is ready (tid=7fc62dced700;cpuset=[9])
03-02-2021  10:49:56.245264 EAL: lcore 3 is ready (tid=7fc62e3f4700;cpuset=[2])
03-02-2021  10:49:56.245358 EAL: lcore 11 is ready (tid=7fc62dbec700;cpuset=[10])
03-02-2021  10:49:56.245667 EAL: lcore 12 is ready (tid=7fc62daeb700;cpuset=[11])
03-02-2021  10:49:56.245771 EAL: lcore 13 is ready (tid=7fc62d9ea700;cpuset=[12])
03-02-2021  10:49:56.246146 EAL: lcore 14 is ready (tid=7fc62d8e9700;cpuset=[13])
03-02-2021  10:49:56.246230 EAL: lcore 2 is ready (tid=7fc62e4f5700;cpuset=[1])
03-02-2021  10:49:56.246616 EAL: lcore 16 is ready (tid=7fc62d6e7700;cpuset=[15])
03-02-2021  10:49:56.246706 EAL: lcore 15 is ready (tid=7fc62d7e8700;cpuset=[14])
03-02-2021  10:49:56.247066 EAL: lcore 1 is ready (tid=7fc62e5f6700;cpuset=[0])
03-02-2021  10:49:56.247602 EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
bash#

bash# cat /etc/*release
ID="wrlinux-small"
NAME="Wind River Linux LTS"
VERSION="10.18.44.16"
VERSION_ID="10.18.44.16"
PRETTY_NAME="Wind River Linux LTS 18.44 Update 16"
bash# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 02:00:17:0a:bf:f9
          inet6 addr: fe80::17ff:fe0a:bff9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:360 (360.0 B)  TX bytes:726 (726.0 B)

eth1      Link encap:Ethernet  HWaddr 02:00:17:0a:87:3c
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 02:00:17:0a:3b:71
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
bash#
bash# ls -l /sys/bus/pci/devices/
lrwxrwxrwx 1 root root 0 Mar  2 04:59 0000:00:03.0 -> ../../../devices/pci0000:00/0000:00:03.0
lrwxrwxrwx 1 root root 0 Mar  2 04:59 0000:00:05.0 -> ../../../devices/pci0000:00/0000:00:05.0
lrwxrwxrwx 1 root root 0 Mar  2 04:59 0000:00:06.0 -> ../../../devices/pci0000:00/0000:00:06.0
bash#
bash# ethtool -i eth0 | grep bus-info
bus-info: 0000:00:03.0
bash# ethtool -i eth1 | grep bus-info
bus-info: 0000:00:05.0
bash# ethtool -i eth2 | grep bus-info
bus-info: 0000:00:06.0
bash#
bash# lspci -v
00:03.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function
        Subsystem: Oracle/SUN Device 16d7
        Physical Slot: 3
        Flags: bus master, fast devsel, latency 0
        Memory at fe300000 (64-bit, prefetchable) [size=16K]
        Memory at fe000000 (64-bit, prefetchable) [size=1M]
        Memory at fe304000 (64-bit, prefetchable) [size=16K]
        Capabilities: [a0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Kernel driver in use: bnxt_en

00:05.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function
        Subsystem: Oracle/SUN Device 16d7
        Physical Slot: 5
        Flags: bus master, fast devsel, latency 0
        Memory at fe30c000 (64-bit, prefetchable) [size=16K]
        Memory at fe100000 (64-bit, prefetchable) [size=1M]
        Memory at fe310000 (64-bit, prefetchable) [size=16K]
        Capabilities: [a0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Kernel driver in use: bnxt_en

00:06.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function
        Subsystem: Oracle/SUN Device 16d7
        Physical Slot: 6
        Flags: bus master, fast devsel, latency 0
        Memory at fe314000 (64-bit, prefetchable) [size=16K]
        Memory at fe200000 (64-bit, prefetchable) [size=1M]
        Memory at fe318000 (64-bit, prefetchable) [size=16K]
        Capabilities: [a0] MSI-X: Enable+ Count=4 Masked-
        Capabilities: [ac] Express Endpoint, MSI 00
        Kernel driver in use: bnxt_en

bash# python3 dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:00:03.0 'NetXtreme-E Ethernet Virtual Function' if=eth0 drv=bnxt_en unused=igb_uio
0000:00:05.0 'NetXtreme-E Ethernet Virtual Function' if=eth1 drv=bnxt_en unused=igb_uio
0000:00:06.0 'NetXtreme-E Ethernet Virtual Function' if=eth2 drv=bnxt_en unused=igb_uio

Other network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other crypto devices
====================
<none>
bash# python3 dpdk-devbind.py -b igb_uio 00:06.0
bash# ifconfig -a

eth0      Link encap:Ethernet  HWaddr 02:00:17:04:f9:13
          inet addr:10.100.0.38  Bcast:10.100.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:859 errors:0 dropped:0 overruns:0 frame:0
          TX packets:783 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:152379 (148.8 KiB)  TX bytes:71516 (69.8 KiB)

eth1      Link encap:Ethernet  HWaddr 02:00:17:07:1b:5b
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

bash# python3 dpdk-devbind.py -b bnxt_en 00:06.0
bash# ifconfig -a

eth0      Link encap:Ethernet  HWaddr 02:00:17:04:f9:13
          inet addr:10.100.0.38  Bcast:10.100.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:870 errors:0 dropped:0 overruns:0 frame:0
          TX packets:792 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:153065 (149.4 KiB)  TX bytes:72030 (70.3 KiB)

eth1      Link encap:Ethernet  HWaddr 02:00:17:07:1b:5b
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 02:00:17:0a:5b:58
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

bash#

dmesg: enter image description here enter image description here

Ravi Kumar
  • 85
  • 1
  • 8
  • `dmesg`? `lspci`? – rkosegi Mar 02 '21 at 05:43
  • lspci has been included, will add dmesg in a sec – Ravi Kumar Mar 02 '21 at 05:45
  • I don't see output of [lspci](https://linux.die.net/man/8/lspci) here – rkosegi Mar 02 '21 at 05:46
  • bash# lspci 00:03.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function 00:05.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function 00:06.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme-E Ethernet Virtual Function – Ravi Kumar Mar 02 '21 at 05:53
  • @Ravi when you bind with userspace the NIC will not be present in kernel. hence `ifcodfig, ethool, ip` will not list the device. IIn DPDK application if you have build with shared library mode you need pass `-d librte_net_bnxt.so`. If this is static library please run DPDK applciation with `--log-level=pmd,8` to further help you – Vipin Varghese Mar 02 '21 at 07:45
  • @Ravi waiting for your updates. – Vipin Varghese Mar 02 '21 at 11:19
  • @VipinVarghese : Sorry for the delay, got stuck with work from morning. I just added the details you asked for in the Edit 1 section of the question. Basically, - librte_net_bnxt.so is loaded. Is there any other PMD to add for this NIC series? – Ravi Kumar Mar 02 '21 at 13:18
  • @Ravi looks like there is DPDK bus probe, which means your application is built in shared library mode. Can you check the same? Can you share the DPDK eal args? and can you please pass option `-d librte_pmd_bnxt.so` – Vipin Varghese Mar 02 '21 at 14:29
  • Can you give me some pointer on how to check if it is in shared library mode? Also, how to obtain EAL args? What does the "-d" option do in specific? – Ravi Kumar Mar 02 '21 at 14:43

1 Answers1

1

Based on the information from the logs, the application is built with shared library mode. Currently, it opens and uses ixgbe and e1000

03-02-2021  10:49:43.406664 EAL: open shared lib /usr/lib64/dpdk-18/librte_pmd_ixgbe.so.2.1
03-02-2021  10:49:43.407327 EAL: open shared lib /usr/lib64/dpdk-18/librte_pmd_e1000.so.1.1

In order to make this work for bnxt PMD, one would need to pass -d librte_pmd_bnxt.so. This can be done by

  1. cmdline arguments parameters if rte_eal_init uses argc and argv
  2. changing the Makefile or meson file to link with -lrte_rte_bnxt.

note: requested for information on DPDK version and application details (is it sample, project, custom). based on the logs, this looks like 18.11

Vipin Varghese
  • 4,540
  • 2
  • 9
  • 25
  • DPDK version is 18.11, its a project that i cannot disclose about. I use rte_eal_init, I will try this option now – Ravi Kumar Mar 03 '21 at 05:22
  • @RaviKumar thanks for confirming it is 18.11 as suspected, if the answer has helped you to understand what was going wrong. Please accept and upvoite. – Vipin Varghese Mar 03 '21 at 05:35
  • I will definitely upvote as soon as this resolves. "-d" is already being sent along with other vendor drivers. – Ravi Kumar Mar 03 '21 at 06:06
  • Can you tell me what logger statements can be used in DPDK librte_* files? i have tried using RTE_LOG, RTE_ETHDEV_LOG and none seem to update the /mnt/disk0/dpdk.log file. Whats the correct way to print some logs? – Ravi Kumar Mar 03 '21 at 09:14