Krping module for RDMA

Question

I have been trying to make krpring module work which can be found here: http://git.openfabrics.org/?p=~sgrimberg/krping.git;a=summary

[  446.242534] rdma_krping: Unknown symbol ib_alloc_fast_reg_page_list (err -22)  
[  446.242542] rdma_krping: disagrees about version of symbol rdma_resolve_addr  
[  446.242542] rdma_krping: Unknown symbol rdma_resolve_addr (err -22)  
[  446.242549] rdma_krping: disagrees about version of symbol ib_reg_phys_mr 
  **and so on...**

I install the driver through Mellanox ofed kernel 3.3. Already copied Module.symvers from ofa-kernel/default but it didn't work.

I have tried it on different kernels on ubuntu 12.04 and 14.04. The kernel that I have used are 3.18, 3.5 and 4.0. However, everytime I get the same issue. If somebody has compiled krping and could help with this? Specially the kernel version, OS and procedure to install the driver they used.

MAKEFILE1

KSRC=/lib/modules/`uname -r`/build
KOBJ=/lib/modules/`uname -r`/build


obj-m += rdma_krping.o
rdma_krping-y           := getopt.o krping.o

default:
    make -C $(KSRC) M=`pwd` modules

install:
    make -C $(KSRC) M=`pwd` modules_install
    depmod -a

clean:
    rm -f *.o
    rm -f *.ko
    rm -f rdma_krping.mod.c
    rm -f Module.symvers
    rm -f Module.markers

MAKEFILE2

KSRC=/lib/modules/$(shell uname -r)/build
KOBJ=/lib/modules/$(shell uname -r)/build


EXTRA_CFLAGS += -DLINUX -D__KERNEL__ -DMODULE -O2 -pipe -Wall
EXTRA_CFLAGS += $(shell echo $(BACKPORT_INCLUDES) | sed -e 's@/var/tmp/OFED_topdir/BUILD@/usr/src@')
EXTRA_CFLAGS += $(shell [ -f $(KOBJ)/include/linux/modversions.h ] && \
            echo "-DMODVERSIONS -DEXPORT_SYMTAB \
                  -include $(KSRC)/include/linux/modversions.h")
EXTRA_CFLAGS += $(shell [ -f $(KOBJ)/include/config/modversions.h ] && \
            echo "-DMODVERSIONS -DEXPORT_SYMTAB \
                  -include $(KSRC)/include/config/modversions.h")

autoconf_h=$(shell /bin/ls -1 $(KSRC)/include/*/autoconf.h 2> /dev/null | head -1)
kconfig_h=$(shell /bin/ls -1 $(KSRC)/include/*/kconfig.h 2> /dev/null | head -1)

ifneq ($(kconfig_h),)
KCONFIG_H = -include $(kconfig_h)
endif

ofa_autoconf_h=$(shell /bin/ls -1 $(OFA)/include/*/autoconf.h 2> /dev/null | head -1)
ifneq ($(ofa_autoconf_h),)
OFA_AUTOCONF_H = -include $(ofa_autoconf_h)
endif

obj-m += rdma_krping.o
rdma_krping-y           := getopt.o krping.o

default:
    -cp -f $(OFA)/Module*.symvers $(TEST_SRC)/Module.symvers
    make -C $(KSRC) O=$(KOBJ) SUBDIRS=$(shell pwd) \
        LINUXINCLUDE=' \
        -D__OFED_BUILD__ \
        $(EXTRA_CFLAGS) \
        -include $(autoconf_h) \
        $(OFA_AUTOCONF_H) \
        $(KCONFIG_H) \
        -I$(OFA)/include \
        $(INCLUDE_COMPAT) \
        $$(if $$(CONFIG_XEN),-D__XEN_INTERFACE_VERSION__=$$(CONFIG_XEN_INTERFACE_VERSION)) \
        $$(if $$(CONFIG_XEN),-I$$(srctree)/arch/x86/include/mach-xen) \
        -I$$(srctree)/arch/$$(SRCARCH)/include \
        -Iarch/$$(SRCARCH)/include/generated \
        -Iinclude \
        -I$$(srctree)/arch/$$(SRCARCH)/include/uapi \
        -Iarch/$$(SRCARCH)/include/generated/uapi \
        -I$$(srctree)/include \
        -I$$(srctree)/include/uapi \
        -Iinclude/generated/uapi \
        $$(if $$(KBUILD_SRC),-Iinclude2 -I$$(srctree)/include) \
        -I$$(srctree)/arch/$$(SRCARCH)/include \
        -Iarch/$$(SRCARCH)/include/generated \
        ' \
        modulesymfile=$(TEST_SRC)/Module.symvers \
        modules

install:
    make -C $(KSRC) O=$(KOBJ) SUBDIRS=$(shell pwd) modules_install
    depmod -a

clean:
    rm -f *.o
    rm -f *.ko
    rm -f rdma_krping.mod.c
    rm -f Module*.symvers

Above are the two makefiles I have tried to compile the module. Suggest any changes that I can try.

Answer 1

The ib_alloc_fast_reg_page_list function has been removed in kernel 4.4. It was replaced with the new ib_alloc_mr as part of the new fast registration API patches.

If you are using MLNX OFED though, it replaces the modules providing these APIs, so it does not matter what kernel version you are using, only the version of MLNX OFED.

Be sure to check that you are compiling your module against the MLNX OFED header files and not against the kernel's. I haven't tried it, but the user manual has an explanation about how to compile Lustre against the MLNX OFED kernel modules, so you may want to try that. It says to use:

EXTRA_LNET_INCLUDE="-I/usr/src/ofa_kernel/default/include/ -include /usr/src/ofa_kernel/default/include/linux/compat-2.6.h" ./configure --with-o2ib=/usr/src/ofa_kernel/default/

I believe this means you would have to add similar flags to your C pre-processor command line. Perhaps there are other necessary changes hidden in their configure script.