Depends on what you call "kernel mode" function. If I understood you correct, you mean that some C functions ask to do something, like malloc()
calls brk()
so kernel would increase process heap size.
So everything you need is detect which functions ask something from kernel which usually done in by calling a System Call such as brk()
.
Now it's time to determine which functions call which system calls. There are two ways to do that:
- Static code analys. Just build a callgraph of a glibc, then plot graphs where root is a syscall. The only problem is complexity of glibc, so it would be hard to build entire callgraph.
- Dynamic code analysis. Attach a debugger/tracer to your process, and when process calls a syscall, print its stack. This approach doesn't cover all functions and cases, but easy to implement.
I.e. using gdb:
myaut@panther:~> gdb /bin/cat
...
(gdb) b brk
Function "brk" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
(gdb) commands 1
> bt
> cont
> end
(gdb) r
Starting program: /bin/cat
...
Breakpoint 1, __brk (addr=0x62d000) at ../sysdeps/unix/sysv/linux/x86_64/brk.c:28
28 in ../sysdeps/unix/sysv/linux/x86_64/brk.c
#0 __brk (addr=0x62d000) at ../sysdeps/unix/sysv/linux/x86_64/brk.c:28
#1 0x00007ffff7b0eea5 in __GI___sbrk (increment=135168) at sbrk.c:53
#2 0x00007ffff7aaeda9 in __GI___default_morecore (increment=<optimized out>) at morecore.c:48
#3 0x00007ffff7aaaa37 in sysmalloc (av=0x7ffff7dd5640 <main_arena>, nb=32) at malloc.c:2454
#4 _int_malloc (av=0x7ffff7dd5640 <main_arena>, bytes=5) at malloc.c:3718
#5 0x00007ffff7aac263 in __GI___libc_malloc (bytes=5) at malloc.c:2859
...
#10 0x0000000000401d8a in main (argc=1, argv=0x7fffffffdd08) at cat.c:563
Or some dynamic tracing engine such as SystemTap (Linux):
# stap -e '
probe syscall.brk {
if(pid() == target())
print_ubacktrace();
} ' -c /bin/cat -d /lib/x86_64-linux-gnu/libc.so.6
...
0x7f5edb427ffa : brk+0xa/0x70 [/lib/x86_64-linux-gnu/libc-2.13.so]
0x7f5edb4280ad : sbrk+0x4d/0xb0 [/lib/x86_64-linux-gnu/libc-2.13.so]
0x7f5edb3d0ae9 : __default_morecore+0x9/0xb0 [/lib/x86_64-linux-gnu/libc-2.13.so]
0x7f5edb3cd1e6 : malloc_trim+0x15e6/0x23c0 [/lib/x86_64-linux-gnu/libc-2.13.so]
0x7f5edb3cec00 : malloc+0x70/0x340 [/lib/x86_64-linux-gnu/libc-2.13.so]
...
0x401bea [/bin/cat+0x1bea/0xc000]
Or DTrace:
root@sol11:~# dtrace -n '
syscall::brk:entry
/pid == $target/ {
ustack()
}' -c '/bin/cat'
CPU ID FUNCTION:NAME
0 58 brk:entry
libc.so.1`_brk_unlocked+0x15
libc.so.1`sbrk+0x38
...
libc.so.1`malloc+0x38
libc.so.1`strdup+0x29
libc.so.1`expand_locale_name+0x333
libc.so.1`setlocale+0x89c
cat`main+0x23
cat`_start+0x7d