2

I wanted to build some sort of API monitor by hooking all ntdll functions using Detours API. Each hooked function will call the original implementation and than add notice for this call inside std base data structure.

However, I encountered a scenario of recursive hooking where my hooked function is indirectly calling itself.

Therefore, I tried to use the Tls memory to set a bit per thread that indicate that current function is called from hooked function, so avoid calling the hook again (execute the original function only).

But my recursive hook guard also uses some calls for memory allocation function like NtAllocateVirtualMemory, and therefore I'm currently avoid hooking those functions.

Perhaps anybody has encountered a similar issue and implemented hook reentrancy guard in a way that doesn't triggered any memory allocation function (which might be impossible since even if you call a new function and your stack memory is insufficient, it should allocate more memory).

thanks

Dominique
  • 16,450
  • 15
  • 56
  • 112
Irad K
  • 867
  • 6
  • 20
  • use TLS is ok solution for this task. so in what is problem ? – RbMm Apr 30 '19 at 13:14
  • Basically I suspect that when calling hooked function indirectly. For example, When the stack is full and we need more space to extend the call stack. – Irad K Apr 30 '19 at 13:21
  • sorry, i not understand your comment. use tls is very simply and clean solution here. `if (TlsGetValue(g_TlsIndex))return ; TlsSetValue(g_TlsIndex, (void*)1); dotask(); TlsSetValue(g_TlsIndex, 0);` – RbMm Apr 30 '19 at 13:23
  • - is your hooked function recursive per se, or it's only getting recursively called via indirection? if it's non-recursive, then you can easily get away via using a local static variable/flag indicating it's already in flight, and clearing such flag on exit – Dmitry Apr 30 '19 at 13:28
  • @RbMm, if stack is full and i'm calling `TlsSetValue(g_TlsIndex, (void*)1);` from within the hooked function, so it might trigger another hooked function when tls hasn't been set yet. Only if `TlsSetValue` was inline so it would be ok. – Irad K Apr 30 '19 at 13:33
  • Dmitry, my hook logic is not calling the function recursively, but i suspect that it may call it indirectly. when setting the TLS to 1 using `TlsSetValue`, it may add another frame to the stack and thus call some hooked function (`NtAllocateVirtualMemory`) where the TLS is still 0 and we have recursive – Irad K Apr 30 '19 at 13:42
  • @IradK - *if stack is full ..* - **any** call fail. so i not understand your point – RbMm Apr 30 '19 at 13:55
  • 1
    however at some condition `TlsSetValue` internally can allocate memory from heap. if you want 100% avoid this - exist undocumented - *RtlPushFrame, RtlPopFrame, RtlGetFrame* - it special design exactly for this, never allocate any memory and never fail. if you ok use this undocumented api - can post solution with this – RbMm Apr 30 '19 at 14:01
  • @IradK, and I presume (b/c of TLS) it's a multi-thread? In such case, if you suspect TLS would mess up with your other hooked function, don't use TLS. E.g.: come up with own indication of being airbone per thread: setup a `static` `std::set` where you store the hook's thread id (indicating it's in flight) - don't forget to put a mutex on it and removing (again under mutex) it on exit. This way you should be safe. – Dmitry Apr 30 '19 at 14:07
  • @RbMm, Yes, that's sound like a great solution (with the undocument methods), can you post it ? – Irad K Apr 30 '19 at 14:10
  • @IradK - can use code like this - https://pastebin.com/S07WqzbT - call `void rectest()` for test – RbMm Apr 30 '19 at 14:33
  • @RbMm, just to make sure, in the solution you provided, where do i write my logic inside `doTask()` ? furthermore, which dll should I add in order to get `RtlPushFrame, RtlPopFrame, RtlGetFrame`. thanks ! – Irad K May 01 '19 at 08:06
  • this api exported by *ntdll.dll* - so add *ntdll.lib* you need insert your logic between `RtlPushFrame` and `RtlPopFrame` - so in my exampla after `MRD m;` – RbMm May 01 '19 at 08:36
  • @RbMm, after using `GetModuleHandle` to load the dll and `GetProcAddress` to get function symbol it worked ! However, I'm not sure i understand how it works besides that instead of using Tls memory we use the stack itself as a frame ... and when calling hooked function we lookup in all previous frames if there was already hooked function call in call stack... Perhaps you can elaborate a little with comments on the code ? thanks ! – Irad K May 01 '19 at 11:07
  • you can simply use *ntdll.lib* - so not need `GetProcAddress` . yes. we add(push) frame to linked list. in frame exist pointer to previous frame and so on. at begin of function we look - are unique frame (by name) already set. if yes - this is reqursive call - just exit. otherwise we push this frame and begin do task. when finished - pop frame. if we will be called reqursive between push and pop frame - we detect frame at begin and just exit. the `void rectest()` show poc – RbMm May 01 '19 at 11:18
  • Hello @RbMm, I have the same problem in my hooking dll. The link to your code is dead. Could you please post it again? Thanks!!! – Leo.Zhou Mar 01 '22 at 14:41

0 Answers0