0

I am trying to run DPDK helloworld example in a custom Linux distribution. I have installed custom Linux in a VMware player. The program is failed due to the fact the NUMA node is not supported. When I try to execute grep NUMA /boot/config-uname -a`, I don't get any results.

On the other hand, I tried the same activity in Ubuntu VM in VMware player. There DPDK helloworld didn't fail. When I try to execute grep NUMA /boot/config-uname -a` , I got details about NUMA configuration.

The other day, VMplayer provided different default NIC cards for Ubuntu and custom Linux. Hence i changed entries in vmx file.

I am just wondering whether different results are due to any setting parameters in custom linux or vmx file settings in the virtual machine.

Would appreicate your help on this to understand the cause of the issue.

I tried the following commands in different VMs

In CentOS VM

cat /etc/centos-release
centOS Linux release 8.2
uname -r
4.18.0
lscpu
NUMA node(s) :1
NUMA node0 CPU(s): 0-3
grep NUMA /boot/Config.x
CONFIG_NUMA=y

In Ubuntu VM
uname -r
4.10.0

lsb_release -a
Ubuntu 16.04
lscpu
NUMA node(s) :1
NUMA node0 CPU(s): 0-3
grep NUMA /boot/Config.x
CONFIG_NUMA=y
numactl --hardware gives output

In Custom Linux

uname -r
3.16.71
**lscpu output don't contain NUMA details**
zgrep NUMA /proc/config.gz
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
**#CONFIG_NUMA is not set**

numactl --show
**No NUMA support available on this system**

In custom Linux source code the following line is there
sources/kernel-headers/linux-3.4.104/arch/x86/configs/x86_64_defconfig
**Config_NUMA=y**

But as per results, it says CONFIG_NUMA is not set.

Would appreciate your inputs whether do I need to update any other part in kernel source code to enable NUMA node.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
vijay
  • 3
  • 2
  • Hi Vijay, based on the skype session and debugging, it looked like more of the the DPDK binary was not cross build for your custom linux target with `sysroot`. Are you still seeing the issue after proper cross build? If yes, I will highly recommend first run sample libnuma API application on your target first. – Vipin Varghese Sep 27 '20 at 07:27
  • Sure Vipin. I am trying to understand very basic things. Thanks for your comments – vijay Sep 27 '20 at 07:56
  • is not your Ubuntu and custom linux both different version. Why not looking into versions of Linux and see how numa nd topology changes are present? – Vipin Varghese Sep 27 '20 at 12:54
  • @Vipin, based on commands I tried, I think cross-compile is not a solution for the issue I face. I agree it's the ideal way to build any package. As you have done cross compile, would it possible to share some inputs on default NUMA settings in kernel code for CONFIG_NUMA so that I can try the same. I have to rebuild kernel with different versions as well. – vijay Sep 28 '20 at 02:35
  • HI VIjay, Are you saying your cross tool chain has no `libnuma` and has kernel config has `CONFIG_NUMA` disabled? Then you can not build an application which needs libnuma. As shared in skype debug, please disable NUMA_HUGEPAGE in dpdk config. Also check `CONFIG_HUGETLB_` in your kernel. `with regard to cross compiling OS there are better articles in the net which helps. I will update it as answer too` – Vipin Varghese Sep 28 '20 at 03:21

1 Answers1

0

VMplayer, Virtual box or KVM-QEMU is not responsible for NUMA behaviour. It depends upon the NUMA configuration which is used to build kernel such as CONFIG_NUMA=y & CONFIG_AMD_NUMA=y. Similarly for huge Pages too CONFIG_HUGETLB_PAGE=y, CONFIG_HUGETLBFS=y

The easiest way to check all these is to use simple application which uses libnuma & mmap for huge pages on the target. Hence as discussed in comments, the recommended way is to always use cross toolchain for the target and build the binary (and not copy the binary built on Ubuntu to sonicwall or custom Linux).

There are many links which can be used as reference for building kernel from scratch

  1. https://www.itdev.co.uk/blog/building-linux-kernel-cross-compiling-beaglebone-black
  2. https://raspberrypi.stackexchange.com/questions/192/how-do-i-cross-compile-the-kernel-on-a-ubuntu-host
  3. https://gist.github.com/lategoodbye/c7317a42bf7f9c07f5a91baed8c68f75

are some of them

Vipin Varghese
  • 178
  • 5