Hook library function using custom method using dlsym

Question

I am recently started looking in hooking into library from C++ code.
There is a slight confusion with the symbol table creation.
Below is my code (Picked from some online resource, I compiled C code with C++)

hook_main.cpp

#include <stdio.h>
#include <malloc.h>
#include <stdlib.h>

int main()
{
  int *p;
  p = (int *) malloc(10);
  free(p);
  return 0;
}

hook_lib.cpp

#include <stdio.h>
#include <stdint.h>
#include <dlfcn.h>

void *malloc(size_t _size)
{
  static void* (*my_malloc)(size_t) = NULL;
  printf("Custom malloc called\n");
  if(!my_malloc)
    my_malloc = dlsym(RTLD_NEXT,"malloc");
  void *p = my_malloc(_size);
  return p;
}

I am compiling both the files using c++, however it doesn't give the desired output. While debugging, I added

#include <iostream>

in hook_lib.cpp and suddenly my symbol table got changed (library started showing the definition of malloc)

Can somebody please put some light on this behavior. Is this something to do with name mangling ?

Take a look at ldd output, your "hook" shared lib should be first in the linking order. — pptaszni, Mar 31 '22 at 15:39
So, the question is, what makes it work when the cpp header is added. — unbesiegbar, Mar 31 '22 at 16:35
Please show your actual compile and link command lines, and state your system and compiler version. — Employed Russian, Apr 02 '22 at 14:18
g++ -fpermissive -o hook_main.cpp; g++ -fpermissive -shared -fPIC hook_lib.cpp -o libhook.so -ldl — unbesiegbar, Apr 03 '22 at 16:12
Run it using 1) ./hook 2) LD_PRELOAD=path_to_libhook.so ./hook — unbesiegbar, Apr 03 '22 at 16:14

score 0 · Answer 1 · answered Apr 02 '22 at 15:03

0

Your hook_lib.cpp doesn't compile. It could be something like this:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

void *malloc(size_t _size)
{
  static void *(*real_malloc)(size_t) = NULL;
  printf("Custom malloc called\n");
  if(!real_malloc)
    *(void **)real_malloc = dlsym(RTLD_NEXT,"malloc");
  void *p = real_malloc(_size);
  return p;
}

answered Apr 02 '22 at 15:03

Lorinczy Zsigmond

1,749
1
14
21

Yes, I had to type the code as my actual code was on a different system from where I could copy directly. static void* (*my_malloc)(size_t) = NULL; is the first line. I have used the flag -fpermissive to compile it – unbesiegbar Apr 03 '22 at 16:12

score -1 · Answer 2 · answered Apr 01 '22 at 03:49

-1

It is happening because of name mangling by C++.

The function name in object file get modified to _Z6malloc which is the mangled C++ name. Now, when I included iostream, maybe it included the chain of headers which provided the extern declaration of malloc.

Essentially, we get the same expected behavior if we just declare

extern "C"
{
  void *malloc(size_t);
}

in hook_lib.cpp

If we inspect the object file after this, the function name stays as malloc and dlsym is able to locate our function.

answered Apr 01 '22 at 03:49

unbesiegbar

471
2
7
19

This is almost certainly the wrong answer -- `` will have `C++`-correct `malloc()` prototype. Oh, I see -- you failed to `#include ` into `hook_lib.cpp` -- that's the real problem. – Employed Russian Apr 02 '22 at 14:16

Hook library function using custom method using dlsym

2 Answers2