Undefined symbol at runtime. Import Python C++ extension

Question

I have a python package (my_python_package), part of which is a C++ extension (my_ext) with a single function (my_ext_func). The extension depends on my C++ library (libmycpp) and my C++ library depends on libarrow. The problem is that I get an error while importing a function from the extension:

$ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from my_python_package.my_ext import my_ext_func
Traceback (most recent call last):
...
ImportError: /usr/local/lib/libmycpp.so: undefined symbol: _ZNK5arrow6Status8ToStringB5cxx11Ev

libmycpp builds and links correctly and i have some C++ executables that work fine with libmycpp.

I made sure that the extension is correctly linked to the arrow library:

ldd my_ext.so | grep arrow
libarrow.so.1200 => /lib/x86_64-linux-gnu/libarrow.so.1200 (0x00007f2f27f18000)

If I go to the directory in ~/.local/lib/python3.8/site-packages/... with my_ext.so, run the python console and try to import it, everything goes smoothly with no error:

$ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from my_ext import my_ext_func
>>>

It is clear that the problem is about python runtime, but I don't know how to fix it. Please help me to do this.

UPD: I also found that if i go to the directory with my_ext module and try importing in it, everything goes smoothly:

$ cd ~/.local/lib/python3.8/site-packages/my_python_package/my_ext
$ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from my_python_package.my_ext import my_ext_func
>>>

But if I import another module that imports my_ext.my_ext_func, I get an error.

UPD 2: When i set LD_PRELOAD=/lib/x86_64-linux-gnu/libarrow.so.1200 before starting python3, everithing is ok. So i'm totally confused.

When you ran `ldd` on `my_ext.so` were you in the same directory as the file? It looks like it from the command. What happens if you run `ldd` on that file from a different directory? What happens if you run `ldd` on `/usr/local/lib/libmycpp.so`? — Pace, Jul 14 '23 at 16:39
@Pace ldd output from another directory is the same, my_ext.so is linked to arrow. Ldd on /usr/local/lib/libmycpp.so says that libmycpp.so not linked to arrow, but it doesn't matter. I've tried to link libmycpp.so to arrow, got the same error on import. — Jerome D Walker, Jul 14 '23 at 23:10
What's the output from `nm -D /lib/x86_64-linux-gnu/libarrow.so.1200 | grep _ZNK5arrow6Status8ToStringB5cxx11Ev` ? — Employed Russian, Jul 16 '23 at 00:33
@EmployedRussian, this: `0000000000733d70 T _ZNK5arrow6Status8ToStringB5cxx11Ev` — Jerome D Walker, Jul 17 '23 at 10:22

score 2 · Accepted Answer · answered Jul 17 '23 at 17:10

So i'm totally confused.

The problem here is that something changes when you change the working directory or use LD_PRELOAD, but stackoverflow is not really productive in performing "live debugging" to figure out what that something is.

In order to learn "how to fish", do this:

Step 0. In your home directory, run this command:

python3 -c "from my_python_package.my_ext import my_ext_func"

This should reproduce the error.

Step 1. Repeat with:

env LD_DEBUG=symbols,bindings LD_DEBUG_OUTPUT=/tmp/my_ext \
  python3 -c "from my_python_package.my_ext import my_ext_func"

This should reproduce the error again, and there should be a (large) /tmp/my_ext.$pid file. That file should mention the _ZNK5arrow6Status8ToStringB5cxx11Ev symbol.

Step 2. Repeat the same command with after changing directory to ~/.local/lib/python3.8/site-packages/my_python_package/my_ext.

Look for differences between the two /tmp/my_ext.... files. In particular, look for different paths to shared libraries being used in the two scenarios.

Repeat again after changing back to $HOME directory and adding LD_PRELOAD=...:

env LD_DEBUG=symbols,bindings LD_DEBUG_OUTPUT=/tmp/my_ext \
  LD_PRELOAD=/lib/x86_64-linux-gnu/libarrow.so.1200 \
  python3 -c "from my_python_package.my_ext import my_ext_func"

That is what I would have done in your situation, and should give you enough fish for a few days ;-)

If that is still not enough, you should probably ask a new (now much more detailed) question with relevant snippets from the debug files produced above.

I caught the fish! Thank you brother! I had pyarrow installed with its own libarrow. This library was loaded first and made it impossible to load the global library. — Jerome D Walker, Jul 17 '23 at 19:15

Undefined symbol at runtime. Import Python C++ extension

1 Answers1