1

I have kcore and I want to get userspace backtrace from kcore. Because some one from our application is making lot of munmap and making the system hang(CPU soft lockup 22s!). I looked at some macro but still this is just giving me kernel backtrace only. What I want is userspace backtrace.

Good news is I have pointer to task_struct.

task_struct->thread->sp (Kernel stack pointer)
task_struct->thread->usersp (user stack pointer) but this is junk

My question is how to get userspace backtrace from kcore or task_struct.

red0ct
  • 4,840
  • 3
  • 17
  • 44
eswaat
  • 733
  • 1
  • 13
  • 31

2 Answers2

0

First of all vmcores typically don't contain user pages. I'm unaware of any magic which would help here - you would have to inspect vm mappings for given task address space and then inspect physical pages, and I highly doubt the debugger knows how to do it.

But most importantly you likely don't have any valid reason to do it in the first place.

So, what are you trying to achieve?

=======================

Given the edit:

some one from our application is making lot of munmap and making the system hang(CPU soft lockup 22s!).

There may or may not be an actual kernel issue which must be debugged. I don't see any use for userspace stacktraces for this one though.

So as I understand presumed issue is excessive mmap + munmap calls from the application.Inspecting the backtrace of the thread reported with said lockup may or may not happen to catch the culprit. What you really want is to collect backtraces of /all/ callers and sort them by frequency. This can be done (albeit with pain) with systemtap.

0

First of all, vmcore is a immediate full memory snapshot, so it contains all pages (including user pages). But if the user pages are swapped out, they couldn't be accessed. So that is why kdump (and similar tools as your gdb python script) focused on kernel debugging functionality only. For userspace debugging and stacktraces you have to use coredump functionality. By default the coredumps are produced when kernel sends (for example) SIGSEGV to your app, but you can make them when you want by using gcore of modifying kernel. Also there is a "userspace" way of making process dump, see google coredumper project

Also, you can try to unwind user stacktrace directly from kcore - but this is a tricky way, and you will have to hope that userspace stacktrace is not swapped out at the moment. (do you use a swap?) You can see __save_stack_trace_user, it will make sense of how to retrieve userspace context

Alex Hoppus
  • 3,821
  • 4
  • 28
  • 47
  • vmcores are created by makedumpfile, which supports skipping various pages, including ones for userspace. Such configuration is the default on e.g. centos. userpace coredumps are of no use in OP's case. Finally, the difficulty is with having the proper address space when reading user content - crash (the debugger) may or may not know how to deal with this, I'm too lazy to check. –  Nov 04 '15 at 15:18
  • Alex your solution of __save_stack_trace_user looks reasonable. Do you know how to get struct stack_trace from task_struct ? – eswaat Nov 04 '15 at 22:07
  • @eswaat why do you need it? Just get struct pt_regs *regs = task_pt_regs(current); and go on – Alex Hoppus Nov 05 '15 at 07:40
  • Alex just get struct pt_regs *regs = task_pt_regs(current); how ? Don't you think I need struct stack_trace *trace to iterate though while as shown in __save_stack_trace_user ? – eswaat Nov 05 '15 at 18:32