dlsym how do I pass the implicit "this" pointer in nonstatic member function?

Question

I want to call a function with dlsym and it is a member function of an object, I have the pointer, but it is not working. The function lives in the main executable, and I am loading it from a shared library. The function is not exported so direct dlsym fails, using gdb break I calculated the offset of this function and other known exported function, so I do dlsym in the exported function and calculate the offset to the other function. I need to call it but I can't pass the args correctly, the first argument should be the "this" implicit pointer since it is a nonstatic member function.

The function definition is this:

_int64 __fastcall gplayer_controller::DebugCommandHandler(gplayer_controller *const this, int cmd_type, const void *buf, size_t size)

my code is this:

/* open the needed object */
  void *handle = dlopen(NULL, RTLD_LOCAL | RTLD_LAZY);
  if(handle == NULL){
    printf("error w/ dlopen\n" );
  }
  

  int (*fptr)(controller *, int, mma *, size_t);
  fptr = (int (*)(controller *, int, mma *, size_t))dlsym(handle, "lua_pushboolean");

  if(fptr == NULL){
    printf("error w/ funcion\n" );

  }
  else{
    printf("found, ptr: %p\n", fptr);
  }

  gobject_imp *pImp = (gobject_imp*)skill->GetPlayer()->GetObject().GetImpl();

  int (*fptr2)(controller *, int, mma *, size_t) = fptr - 5638326;
  printf("ptr calculation...: %p\n", fptr2);

  mma _mma;
  _mma.cmd = 2040;
  _mma.skillid = 15000;
  _mma.level = skill->GetLevel() + 1;

  printf("data controller %p\n",pImp->_commander );

  (*fptr2)(pImp->_commander,2040,&_mma,10);

the first parameter is the "this" pointer, the other 3 are the normal function params

I'm pretty sure name mangling for a member function doesn't result in a symbol name like `gplayer_controller::DebugCommandHandler`. Better check for null before using the result of `dlsym`. You'll probably need to figure out the real symbol name... — fabian, Nov 02 '22 at 23:07
yeah the name is mangled , the fptr is not null, looks like the first parameter which is the "this" pointer is been taken as the second by the function — Alvaro Hernandorena, Nov 02 '22 at 23:17
Probably have to wrap the sucker. Call a generic `static` member function and in the function some how map the controller instance to the object instance you want, then invoke the actual function you want to call on the mapped object. In general if you ever have to cast a function pointer to make it fit, you're signing yourself up for future debugging. — user4581301, Nov 02 '22 at 23:18
Hint: consider that on x86_64 Linux, `sizeof(int (*)(controller *, int, mma *, size_t))` is 8, but `sizeof(int (controller::*)(int, mma *, size_t))` is 16. — Joseph Sible-Reinstate Monica, Nov 03 '22 at 01:56
Also, there may be a bit of an XY problem here. What's your larger goal that you're trying to accomplish by calling a member function through `dlsym`? — Joseph Sible-Reinstate Monica, Nov 03 '22 at 01:58
I need to call that function in order to, level up a skill... — Alvaro Hernandorena, Nov 03 '22 at 02:17
I don't mean why do you need to call the function at all. I mean, why can't you call the function normally, rather than through dlsym? — Joseph Sible-Reinstate Monica, Nov 03 '22 at 02:35
that function is not exported from the main executable, you may ask how I am making dlsym on it since it is not exported. I found other function that is exported, get the pointer with dlsym, calculate the offset to this other function (got the address offset using gdb setting breaks at each function), that is why in my code the function name is demangled, because I skip all the ptr calculation to avoid confusing. — Alvaro Hernandorena, Nov 03 '22 at 02:38
Okay, now I think I may know where the issue is (an issue with how inheritance is implemented). Two further questions to confirm: (1) Is `*pImp->_commander` just a `controller`, or is it another class that derives from `controller`? (2) Is `DebugCommandHandler` a virtual function? — Joseph Sible-Reinstate Monica, Nov 03 '22 at 02:56
*pImp->_commander returns a pointer to the object gplayer_controller, which is the "this", yes, DebugCommandHandler is a virtual function of gplayer_controller, number 25 in vtable according to pahole — Alvaro Hernandorena, Nov 03 '22 at 03:01
Okay, I suspect the problem is that the adjustment to `this` that the "virtual thunk" is supposed to do isn't getting done. I'll do a little bit more investigation into that myself and then try to write up a detailed answer, but that should at least give you something to search for. — Joseph Sible-Reinstate Monica, Nov 03 '22 at 03:09
Does this answer your question? [What is a "virtual thunk" to a virtual function that inherits from a virtual base class?](https://stackoverflow.com/questions/39181152/what-is-a-virtual-thunk-to-a-virtual-function-that-inherits-from-a-virtual-bas) — Joseph Sible-Reinstate Monica, Nov 03 '22 at 03:18
Yep, it looks like you're making the same mistake as the code in the dupe target: directly calling one class's function with a pointer to another class. Normally C++ magically fixes that up in the background, but not when you're using `dlsym` so it can't tell what you're doing. Either find the corresponding thunk and call that instead, or manually offset the `this` pointer you're passing by the right amount instead. — Joseph Sible-Reinstate Monica, Nov 03 '22 at 03:26
Ahh ok I don't understand it much I'm gonna read that post 10 more times, but I think I understand, this gplayer_controller would be the "derivate" class in the link code right? actually I have just look at pahole and it is indeed a subclass of other class " class gplayer_controller : public controller {" — Alvaro Hernandorena, Nov 03 '22 at 03:31
mmm but controller according to pahole does not have that function in it, seems that it is just a function in gplayer_controller — Alvaro Hernandorena, Nov 03 '22 at 03:33
What happens if you change the first argument of `fptr` from `controller *` to `gplayer_controller *`? — Joseph Sible-Reinstate Monica, Nov 03 '22 at 04:11
Have just tried, it crashed in the same part. you know, in gdb when I break in the function name, it says: Breakpoint 1 at 0x5620ca: gplayer_controller::DebugCommandHandler. (9 locations). But if I say break *0x5620ca it does not break at all — Alvaro Hernandorena, Nov 03 '22 at 04:24
I mean when I call that function by the normal means, it breaks with break fun_name but it does not break with *pointer — Alvaro Hernandorena, Nov 03 '22 at 04:25
I got it to work, and posted my own answer, thank you all for your help. — Alvaro Hernandorena, Nov 04 '22 at 15:02

score 1 · Answer 1 · answered Nov 04 '22 at 15:00

I finally got it working, now I can call that member function like if it where a normal one.

The problem with my code, was that the pointer I was getting was wrong, with gdb I got another address with command info frame, but in fact that pointer was also wrong, the function was been called but with all params in 0, after checking in IDA pro , the address gdb was giving me was after the function started, right after the function setted input params.

.text:0000000000562847                 mov     rbp, rsp
.text:000000000056284A                 push    r15
.text:000000000056284C                 push    r14
.text:000000000056284E                 push    r13
.text:0000000000562850                 push    r12
.text:0000000000562852                 push    rbx
.text:0000000000562853                 sub     rsp, 2528h
.text:000000000056285A                 mov     [rbp+this], rdi
.text:0000000000562861                 mov     [rbp+cmd_type], esi
.text:0000000000562867                 mov     [rbp+buf], rdx
.text:000000000056286E                 mov     [rbp+size], rcx
.text:0000000000562875                 mov     rax, [rbp+this]
.text:000000000056287C                 movzx   eax, byte ptr [rax+44h]
.text:0000000000562880                 xor     eax, 1
.text:0000000000562883                 test    al, al
.text:0000000000562885                 jz      short loc_562891
.text:0000000000562887                 mov     eax, 0
.text:000000000056288C                 jmp     loc_56D049

GDB was giving me the address 0000000000562875, right after moving registers with params to rbp.

I recalculated my ptr to the first line of the function (562847) in IDA. And it worked!! I can call the function, the first param I sent as the implicit "this" param works like a charm.

Manually fixing up pointer addresses is going to be extremely non-standard and brittle. Better to write an `extern "C"` function you can discover & call correctly - either as a wrapper, or to return you a valid method pointer — Useless, Nov 04 '22 at 15:06
that function is not exported from the main executable, but I not getting the pointer manually, I am making dlsym to other exported function and calculating the offset from that pointer, so if the main executable changes the address it would be ok, I hope. But it is strange that the addresses of objects in the main executable is always the same, even after restarting, and they match with IDA address also, didn't expect that, I thought that pointer were absolute and not relative to the main executable. — Alvaro Hernandorena, Nov 04 '22 at 15:58

dlsym how do I pass the implicit "this" pointer in nonstatic member function?

1 Answers1