3

I have a python script which eventually calls a binary that requires certain shared libraries. However, I have been running into the following error:

error while loading shared libraries: libmkl_rt.so: cannot open shared object file: No such file or directory

I have narrowed down the cause of this error to the fact that python overwrites LD_LIBRARY_PATH to be the PWD. When executing the binary manually via the bash, everything works properly as this environment variable is properly set. Here are examples of the current behavior:

$ echo $LD_LIBRARY_PATH
/the/proper/paths

$ python
>>> import os
>>> os.environ.get('LD_LIBRARY_PATH')
/the/proper/paths

$ python script.py
/my/present/working/directory

Here, script.py has the same contents as the second example.

Other information asked for:

$ which python
~/.pyenv/shims/python
$ python --version
Python 3.7.3
$ type python
python is hashed (/home/ec2-user/.pyenv/shims/python)
$ realpath `which python`
/home/ec2-user/.pyenv/shims/python

I would like to know if there's a way to see where this variable is getting changed and/or why this would be happening.

I am aware that I could simply hard-code the in the script, but that is a band-aid solution, I would rather know exactly what is happening under the hood.

pppery
  • 3,731
  • 22
  • 33
  • 46
pimler
  • 61
  • 7
  • 1
    Can't reproduce. `LD_LIBRARY_PATH` is correct in both cases. Why `echo` in `$ echo python` in second example? Typo? Is `LD_LIBRARY_PATH` exported? `python --version`? `type python`? `realpath \`which python\``? – dimich Mar 11 '23 at 08:46
  • @dimich yeah that was a typo and I have updated the prompt with the information you asked for. – pimler Mar 12 '23 at 15:52
  • `~/.pyenv/shims/python` is a shell script which invokes another shell script, in my system it is `/usr/share/pyenv/libexec/pyenv`. That script also processes some plugins and hook directories. Likely you problem is not with python itself but with `pyenv` package and it's configuration. – dimich Mar 12 '23 at 16:50
  • You can trace which process changes environment variable, e.g. using [this shell script](https://gist.github.com/dimich-dmb/f25a7ec41d599e0b5e0bd639f24d7001#file-envtrace-sh). In python run something like `subprocess.run(["/path/to/envtrace.sh", "LD_LIBRARY_PATH"])` – dimich Mar 12 '23 at 18:29
  • @dimich does this script allow the tracking of this env var while I run a python script? I found around where it happens, but this would greatly help narrow it down. – pimler Mar 13 '23 at 20:36
  • No, sorry, it can't track change of variable in realtime. It just shows the value of variable in environments of parent processes up to `init` at the moment of invocation. – dimich Mar 13 '23 at 22:30

1 Answers1

1

I am aware that I could simply hard-code the in the script, but that is a band-aid solution, I would rather know exactly what is happening under the hood.

What's happening under the hood is that some code in your script.py (or code which script.py imports) sets the environment variable.

You can run gdb --args python script.py, set a breakpoint on setenv and putenv, and have GDB stop the program when the environment is reset. However, this will stop somewhere in the Python runtime, and figuring out which Python code is running might get tricky.

That said, setting LD_LIBRARY_PATH is not the right way to fix the "binary that requires certain shared libraries" problem in the first place.

A much better solution is to either link the binary with correct -rpath flag, or to fix already-linked binary using patchelf --set-rpath, so the binary runs correctly without LD_LIBRARY_PATH.

Employed Russian
  • 199,314
  • 34
  • 295
  • 362
  • So it seems that the env var gets overwritten after a specific import in my script. However, nowhere in the script I am importing is there a portion that is playing around with environment variables. (I checked that the value was maintained throughout). wondering if this could be a thing with `import`? -- Also, I was using `LD_LIBRARY_PATH` because that is how the executable is shipped and told to be set up - it's not something I wrote. – pimler Mar 13 '23 at 17:53