Linker library path in makefile confusion

Question

I've been programming a while but I still don't fully understand how the linker behaves.

For example, today I downloaded and installed a library that I want to use in my application in Linux. (It was Xerces - for parsing XML files).

I created a makefile and gave it the path to the .so and .a files in my command : -L/usr/local/lib and also told it the name of the library to include : -lxerces-c-3.1.

My application compiled fine but failed during runtime with "cannot open shared object file libxerces-c-3.1.so". Why would this be the case, when I properly give it the path and name in the makefile?

I then added the library path to the LD_LIBRARY_PATH variable in my .bashrc file and then it worked. That's fine , but if I now remove the path to the library in my makefile and don't even include the name of the library , it still works.

I'm confused as to what is going on here. How can it still find the correct library just by assigning the path to the LD_LIBRARY_PATH variable and will only work if I do so? I have read elsewhere to not even use LD_LIBRARY_PATH.

I appreciate any answer for this. The question is a bit long and hopefully not off-topic but I hope others can learn from it too. Thanks

I'd recommend adding the minimum set of rules from your makefile that demonstrates this problem, 10 lines max. As is, we're left speculating on exactly what is in your makefile and if there is an optimal improvement that can be offered. Good luck. — shellter, Jan 20 '16 at 19:49
Check out this post: http://stackoverflow.com/questions/1904990/what-is-the-difference-between-ld-library-path-and-l-at-link-time , where a similar question has been answered — lima.sierra, Jan 20 '16 at 19:49
`-L` and `LD_LIBRARY_PATH` affect two very *different* (but related) operations. Compile-time linking and runtime linking. — Etan Reisner, Jan 20 '16 at 21:26

score 5 · Answer 1 · answered Jan 20 '16 at 20:22

compilation and running are different things. :)

1) A make file contains rules on how to build your application. So when you write a rule like:

    -L/usr/local/lib -lxerces-c-3.1

You are passing options to the linker. The -L option tells the linker to add additional libraries (in this case '/usr/local/lib') to the linker's search path. The -l option names the library that should be linked in.

2) When you go to run an executable, the loader needs to find all the required libraries. For example, on a Linux system you can use the ldd command to see what shared libraries are used. For example on my system:

ldd FEParser
    linux-vdso.so.1 =>  (0x00007ffcdc7c9000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f835b143000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f835ae3d000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f835ac27000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f835a862000)
/lib64/ld-linux-x86-64.so.2 (0x00007f835b447000)

From this, you can see the name of the library that is linked to the left of the '=>' token, and the path to the library to the right. If a library is not found it will show up as missing.

Now in your case, you were able to successfully compile and link your program due to giving the path and library name to use. You were not able to run the program due to the loader not being able to find the library at run time.

You have several options here:

1) you can move the library in a directory that is in the loaders search path.

2) you can modify LD_LIBRARY_PATH which adds additional directories to the loaders search path. Additionally, the directories specified in LD_LIBRARY_PATH will be passed to the linker (it will be appended after all -L flags). You do need to be careful with this, as if you set it globally (say in .bashrc), then it will effect all compilations that you do. You may or may not want this behavior.

3) As others have specified you can use -Wl,rpath=...., but it is deprecated, and I rarely use it.

4) If you need the library installed in an unusual location, you can add a create a file under /etc/ld.so.conf.d that contains, for example (file is yaml.conf):

# yaml default configuration
/opt/yaml/lib

At system boot, the loader reads all the files in this directory and appends the paths to the loader path. If you make a modification to this directory you can use the ldconfig command to cause the loader to reprocess /etc/ld.so.con.d. N.B. I know this works on centOS and Ubuntu flavors of GNU/Linux - can't speak authoritatively on other flavors.

This might be a question for somewhere else, but how does the executable decide whether to use the static library provided or the shared library? — Engineer999, Jan 20 '16 at 21:16
@Engineer999 The application doesn't decide anything. The runtime linker does. With a statically linked library (at compile time) there is no runtime linking of shared objects to be done for that library at all. — Etan Reisner, Jan 20 '16 at 21:28

nsilent22 · Answer 2 · 2016-01-20T20:20:42.847

2

Compile your program with -Wl,-rpath=/usr/local/lib. This way you'll add /usr/local/lib to runtime library search patch of your program and you won't need the LD_LIBRARY_PATH
Warning: since modern dynamic linkers consider rpath as deprecated you can set also runpath (which supersedes it) by specifying -Wl,-rpath=/usr/local/lib,--enable-new-dtags

edited Jan 20 '16 at 20:20

answered Jan 20 '16 at 19:50

nsilent22

2,763
10
14

rpath is obsolete and not to be used. Has been for looooong time. – SergeyA Jan 20 '16 at 19:51
@SergeyA: Could you elaborate? – nsilent22 Jan 20 '16 at 19:53
instead of rpath, one should use runpath - for the reason that the latter can be still overriddent with LD_LIBRARY_PATH, while the former can not. – SergeyA Jan 20 '16 at 19:57
1

@SergeyA: Well, my `man ld` says nothing about runpath, so it can't be obsoleted for looooong time, since my system is not veeeeery old. – nsilent22 Jan 20 '16 at 20:00
@SergeyA could you back up that claim with a reference? I think it's _not_ deprecated – lima.sierra Jan 20 '16 at 20:00
1

@lima.sierra, of course I can. Here is the copy-paste from man ld.so: `(ELF only) Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not exist. Use of DT_RPATH is deprecated.` – SergeyA Jan 20 '16 at 20:01
@nsilent22, from here, for example: http://man7.org/linux/man-pages/man8/ld.so.8.html or any other link. – SergeyA Jan 20 '16 at 20:06
@SergeyA yep, seems you're right! @nsilent22 have a look at `man 8 ld.so` (mine is from 2015-12-28) :) – lima.sierra Jan 20 '16 at 20:09
@SergeyA: Ok, so maybe it's time to upgrade my Ubuntu 12.04 :) – nsilent22 Jan 20 '16 at 20:12
why does runpath, rpath or LD_LIBRARY_PATH only need the path to the directory containing the libraries , and not the exact library specified? – Engineer999 Jan 20 '16 at 20:14
@nsilent22 could you edit your post and add a deprecation warning? – lima.sierra Jan 20 '16 at 20:14

score 1 · Answer 3 · answered Jan 20 '16 at 19:46

1

There is no mystery here. Default library paths for linker (one you call to make your executable file, for example, ld) and runtime linker (one which is responsible for loading shared libraries when you execute your program, for example, ld.so) are different. Runtime linker uses LD_LIBRARY_PATH, while linker uses whatever was configured when ld was build.

In your case, looks like /usr/local/lib is part of one, but not another.

answered Jan 20 '16 at 19:46

SergeyA

61,605
5
78
137

Thanks for that. How does the runtime linker know which library to link tho? LD_LIBRARY_PATH only has the path to the library directory and doesn't name the actual library – Engineer999 Jan 20 '16 at 20:11
@Engineer999, because the name of the actual .so required by your executable is already recorded in the executable. You can run `ldd ` to see which .so's your file needs, and where runtime linker can find them in the given environment – SergeyA Jan 20 '16 at 20:15
How does my executable know about the .so when I don't even give it the information at compile time, only the header file is included? – Engineer999 Jan 20 '16 at 20:20
@Engineer999, because you still have -l rule. – SergeyA Jan 20 '16 at 20:20
but even when I removed that from the makefile, it still found it – Engineer999 Jan 20 '16 at 21:06
@Engineer999, no this is not possible. – SergeyA Jan 20 '16 at 21:15

score 0 · Answer 4 · answered Jan 20 '16 at 20:09

If you're using static linking, all you have to do is tell the linker where your library is at compile/link time. The library (or as much of it as is necessary) gets copied into your executable, and your executable is standalone.

But for various reasons, these days we generally use dynamic linking, not static linking. With dynamic linking, you have to tell the linker where to find the library at compile/link time, and the dynamic linker (ld.so) has to be able to find the library at run time.

If the library was in one of the standard places (/lib, /usr/lib, etc.) there's no problem. But if you linked against a library in a nonstandard place, in general, you have to add that nonstandard place to your LD_LIBRARY_PATH, so that the runtime linker will always be able to find it.

Linker library path in makefile confusion

4 Answers4

Linked