Runtime "symbol lookup error" after compilation and linking of .so

Question

first of all I've searched this one quite a bit and haven't managed to find a question with the same situation.

I'm building a 32 bit plugin as a shared-object on a 64 bit machine and using cpp-redis. cpp-redis built fine and I used a cmake toolchain to force 32 bit, checked the .o files during build and confirmed they are 32 bit.

I compiled and linked my very basic program (consisting of one main source file and an extra lib for talking to my host application since this is a plugin).

Upon running loading the plugin into the host app, I am greeted with:

symbol lookup error: plugins/samp-redis.so: undefined symbol: _ZN9cpp_redis16redis_subscriberC1ERKSt10shared_ptrINS_7network10io_serviceEE

I'm pretty stumped as I've added the cpp-redis dir to the linker path with -L and the library with -l, here's my comp/link lines:

(SDK_DIR is my host app sdk, pretty small and minimal set of .c/h files)

g++ -fpermissive -fPIC -m32 -std=c++11 -c -O3 -w -D LINUX -I$(SDK_DIR) -I$(SDK_DIR)/amx -I/usr/local/include/cpp_redis (source .cpp files...)
g++ -v -Wall -O2 -m32 -fshort-wchar -s -shared -L/usr/local/lib/ -lcpp_redis -o $(OUTFILE) *.o

Which runs fine with no errors, I ran the link stage with -v to check the flags went in properly and it all looks fine.

(added some newlines for readability)

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure
         -v
         --with-pkgversion='Debian 4.9.2-10'
         --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs
         --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++
         --prefix=/usr
         --program-suffix=-4.9
         --enable-shared
         --enable-linker-build-id
         --libexecdir=/usr/lib
         --without-included-gettext
         --enable-threads=posix
         --with-gxx-include-dir=/usr/include/c++/4.9
         --libdir=/usr/lib
         --enable-nls
         --with-sysroot=/
         --enable-clocale=gnu
         --enable-libstdcxx-debug
         --enable-libstdcxx-time=yes
         --enable-gnu-unique-object
         --disable-vtable-verify
         --enable-plugin
         --with-system-zlib
         --disable-browser-plugin
         --enable-java-awt=gtk
         --enable-gtk-cairo
         --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre
         --enable-java-home
         --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64
         --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64
         --with-arch-directory=amd64
         --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
         --enable-objc-gc
         --enable-multiarch
         --with-arch-32=i586
         --with-abi=m64
         --with-multilib-list=m32,m64,mx32
         --enable-multilib
         --with-tune=generic
         --enable-checking=release
         --build=x86_64-linux-gnu
         --host=x86_64-linux-gnu
         --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10) 
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/32/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32/
    :/lib/../lib32/
    :/usr/lib/../lib32/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../
    :/lib/
    :/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-Wall' '-O2' '-m32' '-fshort-wchar' '-s' '-shared' '-L/usr/local/lib/' '-o' 'samp-redis.so' '-shared-libgcc' '-mtune=generic' '-march=i586'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/collect2
     -plugin /usr/lib/gcc/x86_64-linux-gnu/4.9/liblto_plugin.so
     -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
     -plugin-opt=-fresolution=/tmp/cc0Lo5po.res
     -plugin-opt=-pass-through=-lgcc_s
     -plugin-opt=-pass-through=-lc
     -plugin-opt=-pass-through=-lgcc_s
     --sysroot=/
     --build-id
     --eh-frame-hdr
     -m elf_i386
     --hash-style=gnu
     -shared
     -o samp-redis.so
     -s /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.9/32/crtbeginS.o
     -L/usr/local/lib/
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9/32
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32
     -L/lib/../lib32
     -L/usr/lib/../lib32
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../..
     -lcpp_redis amxplugin2.o amxplugin.o main.o
     -lstdc++
     -lm
     -lgcc_s
     -lc
     -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.9/32/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32/crtn.o

Am I misunderstanding the linking stage or statically linking? It has been a while since I did C++... I'm all about Python and Go these days!

EDIT:

Compiling with -W,--no-undefined results in this output:

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure
     -v
     --with-pkgversion='Debian 4.9.2-10'
     --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs
     --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++
     --prefix=/usr
     --program-suffix=-4.9
     --enable-shared
     --enable-linker-build-id
     --libexecdir=/usr/lib
     --without-included-gettext
     --enable-threads=posix
     --with-gxx-include-dir=/usr/include/c++/4.9
     --libdir=/usr/lib
     --enable-nls
     --with-sysroot=/
     --enable-clocale=gnu
     --enable-libstdcxx-debug
     --enable-libstdcxx-time=yes
     --enable-gnu-unique-object
     --disable-vtable-verify
     --enable-plugin
     --with-system-zlib
     --disable-browser-plugin
     --enable-java-awt=gtk
     --enable-gtk-cairo
     --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre
     --enable-java-home
     --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64
     --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64
     --with-arch-directory=amd64
     --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
     --enable-objc-gc
     --enable-multiarch
     --with-arch-32=i586
     --with-abi=m64
     --with-multilib-list=m32,m64,mx32
     --enable-multilib
     --with-tune=generic
     --enable-checking=release
     --build=x86_64-linux-gnu
     --host=x86_64-linux-gnu
     --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Debian 4.9.2-10)
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.9/32/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32/
    :/lib/../lib32/
    :/usr/lib/../lib32/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/
    :/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../
    :/lib/
    :/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-Wall' '-O2' '-m32' '-fshort-wchar' '-s' '-shared' '-L/usr/local/lib/' '-o' 'samp-redis.so' '-shared-libgcc' '-mtune=generic' '-march=i586'
 /usr/lib/gcc/x86_64-linux-gnu/4.9/collect2
     -plugin /usr/lib/gcc/x86_64-linux-gnu/4.9/liblto_plugin.so
     -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
     -plugin-opt=-fresolution=/tmp/ccOq8qY8.res
     -plugin-opt=-pass-through=-lgcc_s
     -plugin-opt=-pass-through=-lc
     -plugin-opt=-pass-through=-lgcc_s
     --sysroot=/
     --build-id
     --eh-frame-hdr
     -m elf_i386
     --hash-style=gnu
     -shared
     -o samp-redis.so
     -s /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.9/32/crtbeginS.o
     -L/usr/local/lib/
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9/32
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32
     -L/lib/../lib32
     -L/usr/lib/../lib32
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9
     -L/usr/lib/gcc/x86_64-linux-gnu/4.9/../../..
     -Bstatic
     --no-undefined
     -lcpp_redis amxplugin.o amxplugin2.o main.o
     -lstdc++
     -lm
     -lgcc_s
     -lc
     -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.9/32/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.9/../../../../lib32/crtn.o
/usr/bin/ld: cannot find
     -lgcc_s
/usr/bin/ld: cannot find
     -lgcc_s
collect2: error: ld returned 1 exit status
makefile:15: recipe for target 'build' failed
make: *** [build] Error 1

Apparently it can't find gcc_s whatever that is? s=static?

I tried searching for gcc libs:

$ find /usr/ -name libgcc*
/usr/lib/gcc/x86_64-linux-gnu/4.9/libgcc_s.so
/usr/lib/gcc/x86_64-linux-gnu/4.9/libgcc_eh.a
/usr/lib/gcc/x86_64-linux-gnu/4.9/x32/libgcc_s.so
/usr/lib/gcc/x86_64-linux-gnu/4.9/x32/libgcc_eh.a
/usr/lib/gcc/x86_64-linux-gnu/4.9/x32/libgcc.a
/usr/lib/gcc/x86_64-linux-gnu/4.9/libgcc.a
/usr/lib/gcc/x86_64-linux-gnu/4.9/32/libgcc_s.so
/usr/lib/gcc/x86_64-linux-gnu/4.9/32/libgcc_eh.a
/usr/lib/gcc/x86_64-linux-gnu/4.9/32/libgcc.a
/usr/lib/gcc/x86_64-linux-gnu/4.9/libgcc_s_32.so
/usr/lib/gcc/x86_64-linux-gnu/4.9/libgcc_s_x32.so
/usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_s.so
/usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_eh.a
/usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc.a
/usr/lib/x86_64-linux-gnu/libgccpp.so.1
/usr/lib/x86_64-linux-gnu/libgccpp.so.1.0.3

Yet doing a ldconfig search shows files named libgcc.so.1:

$ sudo /sbin/ldconfig -p | grep libgcc
    libgccpp.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libgccpp.so.1
    libgcc_s.so.1 (libc6,x32) => /usr/libx32/libgcc_s.so.1
    libgcc_s.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgcc_s.so.1
    libgcc_s.so.1 (libc6) => /usr/lib32/libgcc_s.so.1

Could this be the reason?

And googling the error brings up some results so I'll keep digging...

By default g++ allows to have unresolved symbols when building shared libraries, which then manifest at runtime. Try to link the shared library with "-Wl,--no-undefined" to explicitly force all symbols to be defined. You might be e.g. missing some additional library to link with. — EmDroid, Jan 25 '17 at 10:11
Btw. the symbol seems to be coming from the cpp_redis library, make sure that the library is in LD_LIBRARY_PATH (or in PATH on Windows I guess) when the app is running. It might be that when running the plugin cannot find and load the cpp_redis library. — EmDroid, Jan 25 '17 at 10:13
Thank you @axalis! That part about g++ allowing unresolved symbols was something I was wondering about and couldn't find in the documentation. I'll try that out later as I was confused as to why the link stage ran fine but runtime didn't. — Southclaws, Jan 25 '17 at 15:54
Now I have another question (maybe worth more research now) but I was under the impression that a "static link" to cpp-redis was going to include everything into my .so file because I don't want users of my plugin to need to build/install the cpp-redis project, I'd rather have it all inside one package. I thought this was the case until you mentioned the LD_LIBRARY_PATH and "when the app is running". A point in the right direction would be great! :) — Southclaws, Jan 25 '17 at 15:56
Does the cpp_redis project build both static (\*.a) and dynamic (\*.so) libraries? If it is so and they are in the same location, for GCC linking the dynamic *.so will be picked up in preference by default. You can then either use the lib name explicitly ("libcpp_redis.a" instead of "-lcpp_redis"), rename the static lib to e.g. libcpp_redic-static.a (and use "-lcpp_redis-static") or make preference to the static library: "-Wl,-Bstatic -lcpp_redis". — EmDroid, Jan 25 '17 at 16:50
Yes, I installed cpp-redis only as a static so there's only the "libcpp-redis.a" file in /usr/local/lib. Now compiling with -W,l, I've updated the topic. — Southclaws, Jan 25 '17 at 17:46
You should link your plugin with `-W,--no-undefined` not cpp_redis — Slava, Jan 26 '17 at 13:50
Okay, thank you for all the help and insight (I don't usually work with C++ in this way!) Would it be easier for me to just compile cpp-redis as a dynamic .so and supply that to my users to be loaded alongside my plugin? Seems like a more robust solution... — Southclaws, Jan 27 '17 at 10:52

Slava · Accepted Answer · 2017-01-26T13:53:50.760

2

Even if you build cpp_redis as static library for it to be linkable with shared library it must be compiled with -fPIC flag ie code must be relocatable. Looks like in your case linker silently ignores incompatible code when linking shared lib. You can check if symbols from cpp_redis resolved into your plugin by running nm with your .so file.

According to your update, you should link your plugin with -W,--no-undefined not cpp_redis. Most probably it will fail as symbols from host program would be missing. But you can check if symbols from cpp_redis are listed as missing or not.

edited Jan 26 '17 at 13:53

answered Jan 25 '17 at 18:04

Slava

43,454
1
47
90

If by executable you mean the host app that loads the .so, I don't have control over that. – Southclaws Jan 25 '17 at 18:18
Then that executable is not for use with plugins. Even if you fix this particular issue your plugin will miss functions that defined in executable itself, not it's libraries. – Slava Jan 25 '17 at 18:21
I'm not entirely sure what you mean, the host program has existed for years and has a rich collection of plugins available. If you're saying I can't compile the cpp-redis code into my app I guess I'll just have to tell users to install cpp-redis alongside my plugin right? – Southclaws Jan 26 '17 at 08:57
OK sorry I misunderstand. Updating answer accordingly. – Slava Jan 26 '17 at 13:42

Runtime "symbol lookup error" after compilation and linking of .so

1 Answers1

Linked