2

Problem

To be concrete, I am using cblas for my application. I create a foo.so that uses cblas as a subroutine, and I dlopen('foo.so', RTLD_LAZY).

This is the flags I use to compile(redacted some -I flags):

g++-8 -std=c++17 -shared -O3 -xc++ 
    -ffast-math -fPIC -fopenmp -lcblas - -o /mnt/xxx/python/test.so

This is the error I get upon dlopen():

test.so: undefined symbol: cblas_dgemm

Here we see that cblas_dgemm is included in the .so.

root@cd872c4b85ff:/mnt/xxx/python# objdump -TC test.so | grep cblas
0000000000000000      D  *UND*  0000000000000000              cblas_ddot
0000000000000000      D  *UND*  0000000000000000              cblas_dsyrk
0000000000000000      D  *UND*  0000000000000000              cblas_dgemm
0000000000000000      D  *UND*  0000000000000000              cblas_dgemv

But in ldd it is not listed as a dependency.

root@cd872c4b85ff:/mnt/xxx/python# ldd test.so
    linux-vdso.so.1 (0x00007ffdbb3c4000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f43f23a7000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f43f2177000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f43f1f5f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f43f1b6e000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f43f17d0000)
    /lib64/ld-linux-x86-64.so.2 (0x0000562eb733a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f43f15cc000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f43f13ab000)

I did a test compile and it says that -lcblas was successfully found:

g++-8 -lcblas -Wl,--verbose | grep blas
attempt to open /usr/lib/gcc/x86_64-linux-gnu/8/libcblas.so failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/8/libcblas.a failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libcblas.so succeeded
-lcblas (/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/libcblas.so)
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status

What does work

One important thing is that if I set my environment variable for LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcblas.so, the executable will run correctly. However, this is obviously not The Right Way to get the executable to work. -Wl,rpath= is also not The Right Way, since cblas will reside on different directories (i.e. on OS X it's in /usr/lib/libcblas.so, and in ubuntu it's /usr/lib/x86_64-linux-gnu/libcblas.so, and I'm guessing there will be more unpredictable locations).

What could be the reason? Am I not linking it correctly? Is the dynamic linker not finding the symbols for some reason?

Since some of the information is redacted, please let me know if there is anything that is needed and I'll add it to the information above. Thanks.

EDIT: The symbols do exist inside of libcblas.so:

root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_ddot
000000000000a0a0 T cblas_ddot
root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_dgemm
0000000000005790 T cblas_dgemm
root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_dgemv
00000000000077f0 T cblas_dgemv
root@3841c2760800:/usr/lib/x86_64-linux-gnu# nm -D libcblas.so | grep cblas_dsyrk
00000000000065f0 T cblas_dsyrk

EDIT:

The dependency graph, I believe, looks something like:

main --(dlopens)--> test.so 
    --(includes)--> header-only linear algebra library 
    --(calls cblas subroutine)--> /usr/lib/x86.../libcblas.so
OneRaynyDay
  • 3,658
  • 2
  • 23
  • 56
  • `-Wl-undefined,dynamic_lookup` is this a real flag? Can you provide a [mcve]? – n. m. could be an AI Aug 24 '18 at 06:56
  • @n.m. I cannot provide an example in this case, but I have since removed the flag(and edited the question, it was supposed to be `-Wl,-undefined,dynamic_lookup`). It is a result of an experimentation. I had hoped that I have provided enough information just short of an MVCE in this case. If it is absolutely necessary I will try to reproduce it with no dependencies to redacted code – OneRaynyDay Aug 24 '18 at 06:58
  • It looks like ld doesn't see these symbols in libcblas you are linking with, and so leaves them undefined and doesn't include libcblas.so as a reference. The reason is probably that the symbols are not in libcblas.so itself but in some other shared library referenced by libcblas.so. – n. m. could be an AI Aug 24 '18 at 07:19
  • @n.m. but then why is it that upon setting the path for `libcblas.so` in `LD_PRELOAD`, it runs successfully? I would imagine that if `libcblas.so` is looking for another `.so`, we would need to actually add that `.so`'s path also to `LD_PRELOAD`? – OneRaynyDay Aug 24 '18 at 07:22
  • Because it loads libcblas.so *and* its dependencies. The path to directotry is not needed, the loader can find everything in the default places. – n. m. could be an AI Aug 24 '18 at 07:28
  • Are you sure you are not using --as-needed ld flag? It's the only way for libcblas to be NOT included in libtest.so dependencies. Or perhaps the linker sees libcblas.a and not libcblas.so? – n. m. could be an AI Aug 24 '18 at 07:31
  • I'm going to create an MVCE for this. As for the `--as-needed` ld flag, I'm not sure since I am using cmake for the executable. I will get back to you on this. – OneRaynyDay Aug 24 '18 at 07:37

1 Answers1

1

So - it was because the ordering of my linkage. What I do is I stream in a C++ file (or string) into this g++ compiling process. To stream, one would do:

g++ ... -xc++ - ...

Where -xc++ is telling g++ it's taking in a c++ file(mandatory), and - is the placeholder for the file string. In my compilation, I have:

g++ ... linker_flags ...  -xc++ -

Which means that it actually wasn't getting linked correctly(linker_flags need to be after the file name).

I knew that the ordering is important, but in this situation when I'm streaming in a file, I forgot about this principle and that's why the symbols were undefined.

OneRaynyDay
  • 3,658
  • 2
  • 23
  • 56