3

I am trying to build a kernel module for my DNS-320L NAS.

I have built crosstool-ng in a Debian lenny chroot environment but after I have compiled my kernel module and try and install it, I get:

insmod: error inserting 'kernel/net/ip4/ipip.ko': -1 Invalid module format

When I examine a working kernel module with readelf, I get:

~/ct-ng-build$ readelf -h ~/ct-ng-build/kernel/orig/ipip.ko 
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          6696 (bytes into file)
  Flags:                             0x5000000, Version5 EABI
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         51

But my module has a few differences:

~/ct-ng-build$ readelf -h ~/ct-ng-build/kernel/modules/lib/modules/2.6.31.8/kernel/net/ipv4/ipip.ko 
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 61 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            ARM
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          6280 (bytes into file)
  Flags:                             0x600, GNU EABI, software FP, VFP
  Size of this header:               52 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           40 (bytes)
  Number of section headers:         23
  Section header string table index: 20

My cross_compile environment is:

$ echo $CROSS_COMPILE 
arm-none-eabi-


make CROSS_COMPILE=${CROSS_COMPILE} INSTALL_PATH=~/ct-ng-build/kernel/install INSTALL_MOD_PATH=~/ct-ng-build/kernel/modules INSTALL_FW_PATH=~/ct-ng-build/kernel/firmware

My gcc is compiled:

$  ${CROSS_COMPILE}gcc -v
Using built-in specs.
Target: arm-none-eabi
Configured with: ~/ct-ng-build/targets/src/gcc-4.3.2/configure --build=x86_64-build_unknown-linux-gnu --host=x86_64-build_unknown-linux-gnu --target=arm-none-eabi --prefix=~/x-tools/arm-none-eabi --with-local-prefix=~/x-tools/arm-none-eabi/arm-none-eabi//sys-root --disable-multilib --disable-libmudflap --with-sysroot=~/x-tools/arm-none-eabi/arm-none-eabi//sys-root --with-newlib --enable-threads=no --disable-shared --with-pkgversion=crosstool-NG-1.9.0 --with-arch=armv5te --with-tune=arm926ej-s --disable-__cxa_atexit --with-gmp=~/ct-ng-build/targets/arm-none-eabi/build/static --with-mpfr=~/ct-ng-build/targets/arm-none-eabi/build/static --enable-target-optspace --disable-nls --enable-symvers=gnu --enable-languages=c,c++
Thread model: single
gcc version 4.3.2 (crosstool-NG-1.9.0)

Any ideas how to make the OS/ABI to "UNIX - System V" and the Flags to "0x5000000, Version5 EABI" instead of "Arm" and "0x600, GNU EABI, software FP, VFP" ?

Or is there another problem?

Thanks!

--- EDIT ---- I am specifically trying to target my existing NAS OS:

Kernel  2.6.31.8 #1 armv5tel
C library   gcc-4.3-mt-1.44.0 

# /lib/libc-2.8.so 
GNU C Library stable release version 2.8, by Roland McGrath et al.
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.3.2.

I have only been able to build a crosstool-ng environment so far using crosstool-ng-1.9.0, but it produced the binaries above.

I tried to set -mabi=apcs-gnu as per 3.17.2 ARM Options but the C library would not compile with strange "missing headers" errors (I may try again - possibly something with my chroot environment).

KevinM
  • 567
  • 4
  • 21
  • Just because the original toochain was built with a particular version of crosstool-NG does not mean that you have to do the same. You do want to match the target's type of C library (e.g. glibc or uClibc) and version number, and use kernel headers for the same kernel version. There might be some leeway with the version numbers of the toolchain components, e.g. the gcc compiler and binutils, but crosstool-NG should allow you to specify the versions you want so long as you don't stray too old or too new from that time period. – sawdust Sep 18 '17 at 08:07
  • I've tried compiling crosstool-ng-13.2, but it fails to compile both the gcc 4.3.2 and gcc 4.3.3 shared compiler (and hence final compiler). I will try again to compile 19.0. (which has compiled before - at least the ct-ng build process preserves the log and configuration in the x-tools target directory). – KevinM Sep 19 '17 at 08:15
  • You need to double check that your toolchain is really for EABI, and not OABI. I did a similar `readelf` for a kernel module, and it produced essentially identical results to your *"working kernel module"*. My toolchain that produced the module is for EABI. IOW my answer is backwards, which seems to be confirmed by https://stackoverflow.com/questions/10780648/arm-gentoo-crossdev-with-uclibc-need-oabi-rather-than-eabi . But your toolchain prefix `arm-none-eabi-` contradicts the idea that it is OABI. – sawdust Sep 20 '17 at 05:49
  • I have reverted to crosstool-NG-1.9.0 (as per the eabi toolchain above) targeting "arm-none-linux-gnu", but it now fails to compile the tool chain! I get "internal compiler error" in the C library. I will try again from fresh tarball sources and fresh 1.9.0 arm-none-linux example configuration. – KevinM Sep 20 '17 at 07:44

2 Answers2

0

Build a new toolchain using crosstool-NG, but configure it for OABI rather than EABI.
See Configuring crosstool-NG.

All debian releases (for ARM) upto and including Lenny were OABI.

Ref: https://wiki.embeddedarm.com/wiki/EABI_vs_OABI

Even though your toolchain prefix is arm-none-eabi-, the binary it has produced looks like OABI.
I have OABI binaries that produce identical readelf output, and EABI binaries that have identical readelf output ("UNIX - System V") to what you want.

This question describes a similar but reversed situation to yours, i.e. his toolchain is generating EABI binaries but he needs OABI.

Apparently your toolchain was built to produce OABI but a misleading prefix was used.
You need to build a new toolchain using crosstool-NG, but configure it for EABI rather than OABI.
Newer versions of crosstool-NG actually make configuration for OABI difficult by forcing the selection EABI unless Use obsolete features is enabled.

ADDENDUM

You say "but configure it for OABI rather than EABI.", but how?

Using a crosstool-NG v1.18 installation, I can specify & build
* Linux kernel version 2.6.31.14,
* gcc version 4.3.2,
* binutils version 2.18a 2.19.1a,
* glibc version 2.8.

In order to unselect Target options ---> Use EABI (unselecting means use OABI),
I first have to select Paths and misc options ---> Use obsolete features.

In this version of crosstool-NG, Use EABI is automatically selected.

The description for the Use EABI menu item has:

Set up the toolchain so that it generates EABI-compliant binaries.

If you say 'n' here, then the toolchain will generate OABI binaries.
OABI has long been deprecated, and is now considered legacy.

Since OABI is considered deprecated in this version of crosstool-NG, the config symbol ARCH_ARM_EABI_FORCE is active unless Use obsolete features is specified.

sawdust
  • 16,103
  • 3
  • 40
  • 50
  • I'm trying crosstool-ng 1.4 now. You say "but configure it for OABI rather than EABI.", but how? The only EABI option I have is a single checkbox that defines "# CT_ARCH_ARM_EABI is not set" or "CT_ARCH_ARM_EABI=y" – KevinM Sep 17 '17 at 12:31
  • *"I'm trying crosstool-ng 1.4"* -- Isn't that a rather old version? Seems like you're trying to directly edit the **.config** file . Why aren't you using the menuconfig? See [Configuring crosstool-NG](http://crosstool-ng.github.io/docs/configuration/). – sawdust Sep 17 '17 at 22:07
  • Sorry for giving the wrong impression - I am always using menuconfig but I can not find any setting for OABI. I was trying v1.4 as that was the version used to build the existing kernel ("Linux version 2.6.31.8 (jack@swtest6) (gcc version 4.3.2 (sdk3.3-ct-ng-1.4.1) ) #1 Wed Aug 22 16:55:05 CST 2012"). – KevinM Sep 18 '17 at 04:08
  • Well, if your existing toolchain was built with "CT_ARCH_ARM_EABI=y", then you want the inverse, "# CT_ARCH_ARM_EABI is not set". – sawdust Sep 18 '17 at 07:48
  • Currently I am unable to even build the CT toolchain. Still trying, see comments above. – KevinM Sep 19 '17 at 08:17
  • I will try the versions and the configuration options you specify (I've seen them, so I know what you are referring to). Can you confirm that you can *build the toolchain*, as above, "out of the box" (fresh downloads from the default download locations, etc)? – KevinM Sep 20 '17 at 07:48
  • On my system with CT-NG v1.18.0, binutils 2.18a does not compile (while 2.19.1a does). However with EABI turned off, gcc 4.3.2 does not compile ("internal compiler error: in arm_dbx_register_number") – KevinM Sep 20 '17 at 13:31
  • @KevinM *"On my system with CT-NG v1.18.0, binutils 2.18a does not compile (while 2.19.1a does)"* -- Same here. But you do not want OABI, you want EABI. See revised answer. – sawdust Sep 21 '17 at 05:23
  • I can't get my CT-NG v1.18.0 configured for EABI to compile anymore either "checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.". I will investigate further. – KevinM Sep 21 '17 at 06:11
  • Be sure to `ct-ng clean` between the `ct-ng menuconfig` and `ct-ng build`. – sawdust Sep 21 '17 at 08:40
  • I don't know what was wrong with the RHEL environment where I go the above error, but I have gone back to my CT-NG 1.9.0 with the EABI build. As a final step, I can get the kernel to produce compatible binaries by editing the "arch/arm/Makefile" and removing "-mabi=aapcs-linux" from the "CONFIG_AEABI" section. HOWEVER, I now get "modprobe: can't load module ntfs (kernel/fs/ntfs/ntfs.ko): unknown symbol in module, or unknown parameter". But at least my *original* question has been answered (I think). – KevinM Sep 22 '17 at 08:44
0

I'm extracting some final comments here:

As @sawdust recommends, I am using EABI in the following CT-NG environment:

crosstool-NG version: 1.9.0
gcc 4.3.2
binutils 2.19.1
libc 2.8
gmp 4.3.2
mpfr 2.4.2

In order to get the GCC compiler to compile, I had to do a couple of hacks! Namely:

  • remove res_hconf.c from targets/src/glibc-2.8/nscd (once, after the sources have been extracted)
  • remove res_hconf from the list of objects in the nscd/Makefile
  • create a soft link to stubs-32.h from stubs-.h in ~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi//sys-root/usr/include/gnu (after the build had started).

I just had to download the correct Linux kernel and install it. The missing command was:

make ARCH=arm kirkwood_defconfig

Thereafter I just had to use make ARCH=arm menuconfig to customise things.

The gcc -v says:

Using built-in specs.
Target: arm-unknown-linux-gnueabi
Configured with: ~/ct-ng-build/targets/src/gcc-4.3.2/configure --build=x86_64-build_unknown-linux-gnu --host=x86_64-build_unknown-linux-gnu --target=arm-unknown-linux-gnueabi --prefix=~/x-tools/arm-unknown-linux-gnueabi --with-sysroot=~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi//sys-root --enable-languages=c,c++ --disable-multilib --with-arch=armv5te --with-tune=arm926ej-s --with-float=soft --disable-shared --with-pkgversion=crosstool-NG-1.9.0 --disable-sjlj-exceptions --enable-__cxa_atexit --disable-libmudflap --disable-libgomp --disable-libssp --with-gmp=~/ct-ng-build/targets/arm-unknown-linux-gnueabi/build/static --with-mpfr=~/ct-ng-build/targets/arm-unknown-linux-gnueabi/build/static --enable-threads=posix --enable-target-optspace --with-local-prefix=~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi//sys-root --disable-nls --enable-symvers=gnu --enable-c99 --enable-long-long
Thread model: posix
gcc version 4.3.2 (crosstool-NG-1.9.0)

Finally, to generate SYSV modules, I had to hack the kernel arch/arm/Makefile to remove the -mabi=aapcs-linux setting from the EABI section.

Now the error is:

"modprobe: can't load module ntfs (kernel/fs/ntfs/ntfs.ko): unknown symbol in module, or unknown parameter

Which is for another question :D

I still can't build a working kernel module (maybe because of the hacks!) but it might also be because I don't have a compatible kernel .config for the NAS.

The final changes were:

  • Not use unwind
  • Disable some kernel debugging and tracing
  • use SLAB instead of SLUB

The changes were selected by tracking the dmsg messages.

The resulting compiled kernel modules can now be loaded, if they are compatible with the compiled kernel.

(Note of warning: I got into a little trouble when I started adding iptables modules when I got locked out of my network connection! Be careful! Fortunately the modules are not reloaded at reboot. )

KevinM
  • 567
  • 4
  • 21
  • *"but it might also be because I don't have a compatible kernel .config for the NAS"* -- Not that I've done anything similar, but I was under the impression that the System.map file was salient. – sawdust Sep 25 '17 at 00:56
  • Actually, my build does not produce a System.map (and hence it does not produce a modules.dep, either. I copy one from elsewhere). – KevinM Sep 30 '17 at 12:25
  • In this case, the dmesg provides all the clues. For example, after satisfying various missing symbols (not forcing unwind, etc.): "ieee80211_crypt_tkip: Unknown symbol wireless_send_event" – KevinM Sep 30 '17 at 12:32