1

I reduced my problem using below test codes,

main.cc

#include <iostream>

int main(int argc, const char** argv) {
  void init2();
  init2();
  return 0;
}

2.cc

#include <iostream>

int init2() {
  void init1();
  init1();
  std::cout<<"init2 called\n";
  return 0;
}

1.cc

#include <dlfcn.h>
#include <pthread.h>
#include <stdio.h>
#include <iostream>

typedef FILE* (*FopenFunction)(const char* path, const char* mode);

static FopenFunction g_libc_fopen = NULL;

void init1() {
 g_libc_fopen = reinterpret_cast<FopenFunction>(
          dlsym(RTLD_NEXT, "fopen"));

 std::cout<<"init1: fopen addr:"<<(void*)g_libc_fopen<<"\n";
}

__attribute__ ((__visibility__("default")))
FILE* fopen_override(const char* path, const char* mode)  __asm__ ("fopen");

__attribute__ ((__visibility__("default")))
FILE* fopen_override(const char* path, const char* mode) {
  return g_libc_fopen(path, mode);
}

Compiled 1.cc into a lib1.so and 2.cc into lib2.so like below,

g++ 1.cc -shared -ldl -fvisibility=default -fPIC -o lib1.so -L.
g++ 2.cc -shared -ldl -fvisibility=default -fPIC -o lib2.so -l1 -L.
g++ main.cc -l2 -l1 -L.

Above steps will produce lib1.so, lib2.so and a.out. The problem here is while running the executable a.out, it is unable to lookup the original "fread" symbol when using dlsym(RTLD_NEXT).

The output is,

arunprasadr@demo:~/works/myex/c++/rtdl_next$ LD_LIBRARY_PATH=./ ./a.out
init1: fopen addr:0
init2 called

But if the change the link process of lib2.so(like below), it seems to be working

g++ 2.cc -shared -ldl -fvisibility=default -fPIC -o lib2.so -L.
g++ main.cc -l2 -l1 -L.
LD_LIBRARY_PATH=./ ./a.out

output:

arunprasadr@demo:~/works/myex/c++/rtdl_next$ LD_LIBRARY_PATH=./ ./a.out
init1: fopen addr:0x7f9e84a9e2c0
init2 called

Can anyone please explain what is happening in the background? Thanks in advance.

Arunprasad Rajkumar
  • 1,374
  • 1
  • 15
  • 31

1 Answers1

3

This is an interesting (and unexpected for me) result.

First, using your original commands, I observe:

LD_DEBUG=symbols,bindings LD_LIBRARY_PATH=./ ./a.out |& grep fopen
     10204: symbol=fopen;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
     10204: symbol=fopen;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     10204: symbol=fopen;  lookup in file=/lib/x86_64-linux-gnu/libgcc_s.so.1 [0]
     10204: symbol=fopen;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
init1: fopen addr:0

Compare this with the with the same output, but removing -l1 from lib2.sos link line:

LD_DEBUG=symbols,bindings LD_LIBRARY_PATH=./ ./a.out |& grep fopen
     10314: symbol=fopen;  lookup in file=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0]
     10314: symbol=fopen;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
     10314: binding file ./lib1.so [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `fopen'
init1: fopen addr:0x7f03692352c0

The question then is: why isn't the loader searching libc.so.6 for fopen in the first case?

The answer: the loader has a linear list of libraries in the _r_debug.r_map link chain, and for RTLD_NEXT will search libraries after the one that is calling dlopen.

Is the order of libraries different for case 1 and case 2? You bet:

case 1:

LD_LIBRARY_PATH=./ ldd ./a.out
    linux-vdso.so.1 =>  (0x00007fff2f1ff000)
    lib2.so => ./lib2.so (0x00007f54a2b12000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f54a27f1000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f54a2430000)
    lib1.so => ./lib1.so (0x00007f54a222e000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f54a1f32000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f54a2d16000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f54a1d1b000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f54a1b17000)

case 2:

LD_LIBRARY_PATH=./ ldd ./a.out
    linux-vdso.so.1 =>  (0x00007fff39fff000)
    lib2.so => ./lib2.so (0x00007f8502329000)
    lib1.so => ./lib1.so (0x00007f8502127000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8501e05000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8501a45000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8501841000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8501544000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f850252d000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f850132e000)

It should now be clear, that for case 2 libc.so.6 follows lib1.so, but for case 1 it does not.

I do not yet understand what causes this particular ordering though. I'll have to think some more about it.

Employed Russian
  • 199,314
  • 34
  • 295
  • 362