Error implementing system call in Linux 5.7

Question

Here's what I did.

In path ./linux-5.7

mkdir hello

cd hello

vim hello.c

#include <linux/kernel.h>

asmlinkage long sys_hello(void) {
    printk ("\n\nHello Kernel\n\n");

    return 0;
}

vim Makefile

obj-y := hello.o

vim arch/x86/entry/syscalls/syscall_64.tbl

439 common  hello           sys_hello

vim include/linux/syscalls.h

asmlinkage long sys_hello(void);

...

Compile the kernel

make -j4

Error ...

ld: arch/x86/entry/syscall_64.o:(.rodata+0xdb8): undefined reference to `__x64_sys_hello'
make: *** [Makefile:1113: vmlinux] Error 1

What's wrong here... ...

...

Does this answer your question? [Unable to add a custom system call on x86 ubuntu linux](https://stackoverflow.com/questions/54840772/unable-to-add-a-custom-system-call-on-x86-ubuntu-linux) — Joseph Sible-Reinstate Monica, Jun 03 '20 at 02:17
First of all, the syntax of defining a system call has changed to : SYSCALL_DEFINEn(...). Check this documentation: [link](https://www.kernel.org/doc/html/v4.13/process/adding-syscalls.html) — losnihciL, Nov 23 '21 at 11:17

Mint Bee · Answer 1 · 2021-11-06T13:54:33.490

I experienced similar problem, but the solution from Joseph's comment didn't work out for me. So, this answer is for those who tried Joseph's one but had not made progress.

Here's steps that I took based on Kernel document

After you put __x64_ in front of the entry point, if you got an error such as

ld: arch/x86/entry/syscall_64.o:(.rodata+0xdc8): undefined reference to `__x64___x64_sys_my_syscall'
ld: arch/x86/entry/syscall_x32.o:(.rodata+0xdc8): undefined reference to `__x64___x64_sys_my_syscall'
make: *** [Makefile:1179: vmlinux] Error 1

then rather then put __x64_ in front of entry point of your syscall, put syscall into both of your arch/x86/entry/syscalls/syscall_64.tbl and arch/x86/entry/syscalls/syscall_32.tbl without any prefix such as

- At the `arch/x86/entry/syscalls/syscall_64.tbl`,
441 common  my_syscall  sys_my_syscall
- At the arch/x86/entry/syscalls/syscall_32.tbl`
441 i386 my_syscall sys_my_syscall

After this, if you still have same problem, then add your syscall into general syscall table: syscall list that shared among several architectures.

Put following in include/uapi/asm-generic/unistd.h:

#define __NR_my_syscall 441
__SYSCALL(__NR_my_syscall, sys_my_syscall)

Note that the kernel version I used is WSL2-Linux-Kernel-linux-msft-wsl-5.10.60.1 and it's built in x64 architechture.

I'm quite novice in OS and linux kernel, so this is just what worked for me and I don't know the exact mechanisim of it; sorry for no mechanisim explanation.

Error implementing system call in Linux 5.7

1 Answers1