1

When using CMake for crosscompiling ARM executables and setting the CMAKE_SYSROOT variable in a toolchain file, the linker does not automatically find libraries which other libraries (which were explicitly linked to) depend on, such as z and glib. When not using a toolchain file (and thus not cross-compiling) these do not need to be explicitly mentioned in target_link_libraries(), however when using the toolchain file they do or else there is always a linker error that they could not be found. I would like to understand why this is.

Brief description of my environment:

  • Using Linux Ubuntu 16 to compile for arm target running Ubuntu 18
  • Root file-system zipped from target and unzipped on development system
  • Linaro cross-build toolchain (7.4.1)
  • a toolchain file for cmake, which defines: CMAKE_SYSROOT, CMAKE_CXX_COMPILER, CMAKE_C_COMPILER and CMAKE_AR_COMPILER
  • Path to compiler: /home/brian/workspace/ddpx_xcompile/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu
  • Path to target root file system: /home/brian/workspace/ddpx_xcompile/ddpx-rootfs-05172019

I've tried many combinations of: setting the CMAKE_LIBRARY_PATH, link_directories etc. However, the only successful option seems to be adding the following inside the target_link_libraries() command: z dl glib-2.0 pcre)

Below is an example of the error message:

<path-to-toolchain>/aarch64-linux-gnu/bin/ld:
 warning: libdl.so.2, needed by <path-to-sharedlib>/lib/libopencv_core.so.4.0.1, not found (try using -rpath or -rpath-link)

And this is followed by a short list of undefined functions which are within the indicated library. This error is for libdl.so.2 not found, there are others.

My error output:

/home/brian/workspace/ddpx_xcompile/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/7.4.1/../../../../aarch64-linux-gnu/bin/ld: warning: libdl.so.2, needed by /home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1, not found (try using -rpath or -rpath-link)
/home/brian/workspace/ddpx_xcompile/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/7.4.1/../../../../aarch64-linux-gnu/bin/ld: warning: libpthread.so.0, needed by /home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1, not found (try using -rpath or -rpath-link)
/home/brian/workspace/ddpx_xcompile/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/7.4.1/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1, not found (try using -rpath or -rpath-link)
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `pthread_key_create@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `gzeof'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `pthread_setspecific@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `pthread_join@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `pthread_getspecific@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `pthread_key_delete@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `pthread_create@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `dlclose@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `gzrewind'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `dlopen@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `dladdr@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `gzopen'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `dlsym@GLIBC_2.17'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `gzclose'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `gzgets'
/home/brian/workspace/test/crossbuild/test_rootfs/ddpx-rootfs-05172019/home/nvidia/3rd_party_libs/opencv/install-cxx11/lib/libopencv_core.so.4.0.1: undefined reference to `gzputs'
collect2: error: ld returned 1 exit status
CMakeFiles/imshow.dir/build.make:95: recipe for target 'imshow' failed
make[2]: *** [imshow] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/imshow.dir/all' failed
make[1]: *** [CMakeFiles/imshow.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

My Cmake Toolchain file:

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

set(TARGET_FS "/home/brian/workspace/ddpx_xcompile/ddpx-rootfs-05172019"  CACHE STRING "test" FORCE)
set(CMAKE_SYSROOT ${TARGET_FS})

#Tried with and without the ${TARGET_FS} prefix, also tried with all uncommented and all commented
#link_directories(/lib/aarch64-linux-gnu)
#link_directories(/usr/lib)
#link_directories(/usr/lib/aarch64-linux-gnu)
#link_directories(${TARGET_FS}/lib/aarch64-linux-gnu)
#link_directories(${TARGET_FS}/usr/lib)
#link_directories(${TARGET_FS}/usr/lib/aarch64-linux-gnu)

#Did not try too much with these
#set(CMAKE_FIND_ROOT_PATH ${CMAKE_SYSROOT})
#set(CMAKE_FIND_ROOT_PATH ${CMAKE_FIND_ROOT_PATH} ${CMAKE_SYSROOT}/lib/aarch64-linux-gnu)

#Tried with two different compiler versions
set(CROSSBIN /home/brian/workspace/ddpx_xcompile/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu)
#set(CROSSBIN /home/brian/workspace/ddpx_xcompile/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu)

set(CMAKE_CXX_COMPILER "${CROSSBIN}/bin/aarch64-linux-gnu-g++")
set(CMAKE_C_COMPILER   "${CROSSBIN}/bin/aarch64-linux-gnu-gcc")
set(CMAKE_AR_COMPILER  "${CROSSBIN}/bin/aarch64-linux-gnu-ar")

#This was also an experiment, no change
#set(CMAKE_PREFIX_PATH ${TARGET_FS})
#list(APPEND CMAKE_PREFIX_PATH ${TARGET_FS}/lib/aarch64-linux-gnu)
#list(APPEND CMAKE_PREFIX_PATH /lib/aarch64-linux-gnu)
#list(APPEND CMAKE_LIBRARY_PATH "/lib/aarch64-linux-gnu")
#list(APPEND CMAKE_LIBRARY_PATH "/usr/lib/aarch64-linux-gnu")

#Tried with the 4 below uncommented or just commented
#set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
#set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
#set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
#set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

message(STATUS "CMAKE_LIBRARY_PATH: ${CMAKE_LIBRARY_PATH}")
yacc
  • 2,915
  • 4
  • 19
  • 33
dwyer2bp
  • 49
  • 1
  • 4
  • The libraries `libdl` and `libpthread` are normally shipped with a compiler and its standard library. Where do you have these libraries (with the **exact** names as in the error messages: `libdl.so.2` and `libpthread.so.0`)? – Tsyvarev Jun 03 '19 at 18:29
  • @Tsyvarev `/home/brian/workspace/ddpx_xcompile/ddpx-rootfs-05172019/lib/aarch64-linux-gnu/libdl.so.2` Also, my toolchain is at: `/home/brian/workspace/ddpx_xcompile/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu` – dwyer2bp Jun 03 '19 at 18:39
  • @Tsyvarev: I did also find an additional libdl.so.2 in the compiler's directory: `/home/brian/workspace/ddpx_xcompile/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/aarch64-linux-gnu/libc/lib/libdl.so.2` – dwyer2bp Jun 03 '19 at 18:58
  • My theory: When compiling for the host machine (no toolchain file) I have to specify a different path to the dependency OpenCV. The linker is able to resolve the location of OpenCV library dependencies (such as dl & z). When I cross-compile, the path to the OpenCV library is within the target root file system. Is the linker just not able to find the path to OpenCV dependencies (dl & z in this simple case)? In this case, am I doing something wrong by 'zipping up' the root file system and then unzipping on the host machine? – dwyer2bp Jun 03 '19 at 20:39
  • All locations you show are **outside of sysroot** you use in your toolchain. That is why the linker doesn't search there. – Tsyvarev Jun 03 '19 at 20:41
  • @Tsyvarev: I know that is misleading, however the path in the cmake toolchain file is to a symbolic link to the right directory. I actually just reverted to no symlinks, and there is absolutely no change. – dwyer2bp Jun 04 '19 at 01:22
  • Please, add **more detailed description** of the paths you use into the question post. The linker cannot find libraries which are part of the standard library support - this **problem** is mainly **about the paths**. Aside from setting the right compiler and right sysroot CMake can do a little about that (all additional commands you use and all additional variables you set are complitely irrelevant to the linker's error). You may also try to compile hello-world example using only command line (without CMake). – Tsyvarev Jun 04 '19 at 07:54
  • Hello world works just fine, it is when I bring in *.so files which depend on other *.so files. I don't know what good changing the paths in the questions post would do for anyone, the paths are essentially identical (either the rootfs is direct path or a symbolic link). I searched for the libdl.so.2 you asked for, and found 2 instances and indicated both paths in the comments here. The problem is with the linker yes, but why?... – dwyer2bp Jun 04 '19 at 13:39
  • I'm having the same issue. @Tsyvarev Where does the linker (i.e. the linker from the linaro toolchain) search? I also have my libraries inside "ddpx-rootfs" and the linker can't find them. Then where is the linker searching? I've seen this post: https://sysprogs.com/w/fixing-rpath-link-issues-with-cross-compilers/ but **I can't understand where exactly should the etc/ld.so.conf file be**, inside the linaro toolchain folder. Do you have a specific answer here? (i.e. exact path where this file should exist) – user1011113 Nov 08 '19 at 14:20
  • @user1011113: Sorry, I cannot help with that. The current question post seems valid, but I cannot help with it too. – Tsyvarev Nov 08 '19 at 14:38

0 Answers0