Edit: I appear to have been mistaken, the backtrace works wonderfully from anywhere on Linux -- it is only when remote debugging from gdb on ubuntu to remote windows that the stacktrace gets absolutely destroyed after entering one of the memory allocation functions in msvcrt... dammit microsoft.
And this happens for both 64bit and 32bit windows, so I'm not sure this is related to the unwind information...
Edit: It appears adding -g3 and -Og has helped with part of the issue in some programs but the problem still persists in other programs, cannot post their source here as it is IP of my company -- sorry!
Background
I am using gcc to compile ubuntu->ubuntu and mingw to compile ubuntu->windows.
I have created a cross platform (linux + windows) memory tracking & leak detection library which hooks malloc/calloc/realloc/free with an assembly bytepatch on the first instructions (not IAT/PLT hooking).
The hook redirects to a gate which checks if the hooks are enabled in the current thread and redirects to the memory tracking hook function if they are, otherwise it just redirects to the trampoline of the real function if they are disabled for that thread.
The library works great and detects leaks on linux/windows (probably would work on mac but I don't have one).
I use the library to programmatically detect leaks from within my code, I can install callbacks on the memory allocation routines and programmatically raise breakpoints (by looping and waiting for debugger to attach then executing asm("int3")) inside the callbacks so that I can attach to my program while it's inside of a call that leaks memory.
Everything works great up until I try to view a backtrace from within my callback, I understand this is is probably because the unwind information is probably not matching my stack anymore because I have inserted new frames and data via the hook routines I have inserted.
Edit: If I am mistaken about the unwind info mismatching the stack being the cause of the incorrect backtrace then please correct me!
The Question
Is there any small hacks I can do to trick GDB into correctly rebuilding the backtrace from within my hook callbacks?
I understand that I can manually walk and edit the unwind info with libdwarf or something but I imagine that would be incredibly cumbersome and large.
So I am wondering if perhaps there is a hack or a cheat I can do which would trick GDB into properly rebuilding the backtrace?
If there are no easy hacks or tricks then what are all of my options for fixing this issue?
Edit: Just to clear up the exact call order of everything:
program
V
malloc
V
hook_malloc -> hooks are disabled -> return malloc trampoline -> real malloc > program
V
hooks are enabled
V
Call original malloc -> malloc trampoline -> real malloc -> returns to hook
V
Record memory size/info etc from malloc
V
Call user defined callback -> **User defined callback* -> returns to hook
V
return to program
It is the "User Defined Callback" where I want to capture a backtrace
