4

the title may be a bit too short to be clear enough.

We have a complex C/C++-project which is built and linked in a lot of separate targets as static libraries. So my problem is that the target_link_libraries does transitive linking but in a wrong order. I need another order since we use PIMPL and the Implementations do not have any headers to include. they just include the headers of the lib which defines the PIMPL-Headers and declared the Impl-object. But for a correct linking they need to be linked after the lib which provides the header for the Pimpl. This does not happen.

This one scenario only occurs if we are building with a gcc-compiler and linker which we use for our unit-test-framework. For our target-project we use a TI clang-toolchain which provides the linker-option --reread_libs which automatically solves this problem. This is not available in gcc as far as I know.

So now to Cmake in a bit more detail. The project is too big to make a minimal example but I will try to describe the problem and shrink it to the problem:

So we have multiple static-libs inside one module:

# static libs inside the platform
add_library(memutils memutils/ForwardDeclaredStorage.hpp)
target_include_directories(memutils INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}/memutils)

set_target_properties(memutils PROPERTIES LINKER_LANGUAGE CXX)

# contains the Pimpl-Headers to use, here task and mutex as extract
add_library(osal osal/mutex.hpp osal/task.hpp osal/TaskManager.hpp osal/TaskManager.cpp) 
target_include_directories(osal PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/osal)

target_link_libraries(osal PUBLIC memutils)

add_library(system sys/system.hpp sys/system.cpp)

target_link_libraries(system PUBLIC osal)

target_include_directories(system PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/sys)
add_library(platformlib INTERFACE)
# example with INTERFACE keyword. Adding a dummy source and using PUBLIC does not change anything
target_link_libraries(platformlib INTERFACE osal system memutils)

# some extensions
add_library(additionslib add/additions.hpp add/additions.cpp )

target_link_libraries(additionslib PUBLIC platformlib)

target_include_directories(additionslib PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/add)

# the Impl-Lib which needs the headers
add_library(gcc_impl STATIC impls/MutexImpl.cpp impls/TaskImpl.cpp)
# so the implementations get the headers
target_link_libraries(gcc_impl PUBLIC platformlib)

# the test-case. its a project!
project(tests)
add_executable(tests test.hpp test.cpp main.cpp )

# linking all the other libs, also the test-framework (gtest), PRIVATE, PUBLIC... doesnt matter. In this case the extensions will link the platformlib transitive.
target_link_libraries(tests PUBLIC additionslib gcc_impl)

So how does the link order look? What CMake produces as a resulting link-order here is:

first it is like a "top-level-linking" and after this follow all transitive libs follow:

libadditionslib.a 
libgcc_impl.a 
// now the transitive libs in order of top-level-linked libs
libsystem.a 
libosal.a 
libmemutils.a

Now we have Problem. The platform needs the gcc_impl but it linked before the platform. I need it to be linked after the platform. If I change the generated linker-command and add the impl at last link it works fine. If not I get unreferenced symbols to the impls since they are missing then.

I tried a lot. I fiddled around with link-keywords INTERFACE PUBLIC PRIVATE in every lib. Tried to include just headers of the platform via generators instead of linking it directly. I also tried to directly link the osal which has the headers with the gcc-impl:

target_link_libraries(tests PRIVATE additionslib osal gcc_impl)

Guess what happens?

libadditionslib.a 
libosal.a 
libgcc_impl.a 
// now the transitive libs in order of top-level-linked libs
libsystem.a 
libosal.a 
libmemutils.a

The effect is that the error is still the same since the system lib also needs the platform which contains the osal which needs the impl linked after it.

But no matter what I cannot change this "level"-Order of linking and force the gcc_impl to occur last.

What I need is a "in-place" linking and not a level-base linking:

libadditionslib.a 
libsystem.a 
libosal.a 
libmemutils.a
libgcc_impl.a // gccImpl after all the platform-links!

How can I achieve this last one?

I added a minimal example that produces this linkage-error (undefined reference): https://godbolt.org/z/d19Wefrhx

you can also see that link-order:

/opt/compiler-explorer/gcc-11.3.0/bin/g++ -fdiagnostics-color=always -std=c++14 -O2 -g -L/app -Wl,-rpath,/app -Wl,-rpath,/opt/compiler-explorer/gcc-11.3.0/lib64 -Wl,-rpath,/opt/compiler-explorer/gcc-11.3.0/lib32 CMakeFiles/tests.dir/test.cpp.o CMakeFiles/tests.dir/main.cpp.o -o tests  libadditionslib.a libgcc_impl.a libsystem.a libosal.a libmemutils.a

unfortunately I can't directly execute command-lines in godbolt. But I am sure if libgcc_impl.a occurs again as last one it would work. At least that's exactly how it works for us here. How can I force CMake to link the impl-library at the end?

NetoBF
  • 127
  • 6
  • 1
    if `platform` depends on `gcc_impl` why `target_link_library(gcc_impl platformlib)` rather than `target_link_library(platformlib gcc_impl)`? – Alan Birtles Aug 24 '23 at 11:59
  • @AlanBirtles because the platform doesn't know about the gcc_impl and should not know. There are also other impls for other hardwares. The platformlib just provides the headers for those impls and of course uses them as well as well as all other platform-linking entitites. Or do you mean I should try this from the "outside" at the test-project-level? – NetoBF Aug 24 '23 at 12:10
  • What your ideal model of how things works isn't relevant to the linker, if a library needs to use another library, it needs to link to that library – Alan Birtles Aug 24 '23 at 12:12
  • @AlanBirtles so this means there is no way to implement a good abstraction for PIMPLs with Cmake without placing all the knowledge of all linkable Impl-possibilites inside the platform? Well this is really really bad. We need to evaluate other build-systems then. Thanks so far! – NetoBF Aug 24 '23 at 12:17
  • 1
    Unless I'm misunderstanding something, I don't think cmake is the problem, if you need to link to a library you need to tell cmake to link to that library? the same would apply with any other build system. Maybe a [mre] would help illustrate your problem. You've told cmake that `gcc_impl` depends on `platform` so it is deliberately linking the libraries in the right order to make that work – Alan Birtles Aug 24 '23 at 12:20
  • @AlanBirtles No I think you are understanding that correctly. PIMPL affords that it has headers which only declare the concrete implementation. And in only(!) cpp-files there would be the implementation which need those header-declarations. So it is a "source-file-only" lib if you want to see it like this in contrary to "header-only". Now the impl needs the headers, yes. so it links right at this point. But, e.g. the extensions need the platform and thus this will be linked all after in the transitive chain where the impls won't be linked again. And so the definition-ref is missing. – NetoBF Aug 24 '23 at 12:35
  • But honestly that is how you set up a PIMPL-design. Source-files are always separate and MUST be exchangable with other Impls so you have a stable ABI and headers are somewhere else. I'm really curious how others solve this problem with CMake? Because Pimpl everytime affords to have the implementations linked last where you need them. and the lib with the headers is not allowed to know about the implementation. – NetoBF Aug 24 '23 at 12:37
  • 3
    I still think we need a [mre], I still don't understand what you're trying to do – Alan Birtles Aug 24 '23 at 12:39
  • I added a minimal example. This reproduces the behaviour. And no matter if I may have set an include wrong or anything. That's not the matter. It's about that we need the impls get linked last, since this is the real problem we face. – NetoBF Aug 25 '23 at 11:01
  • `I added a minimal example that produces this linkage-error (undefined reference)` +100 – KamilCuk Aug 25 '23 at 11:02
  • Note that a [mre] should be self contained within the question without relying on external links – Alan Birtles Aug 25 '23 at 11:20

2 Answers2

3

So generally what you need to do is to link twice with both the interface and the implementaiton. Either you have to have cyclic dependency between libraries, in which case LINK_INTERFACE_MULTIPLICITY kicks in, or link twice.

You can have cyclic dependency by linking system or osal or memutils with gcc_impl. Typically what I would see is to another layer of abstraction:

target_link_libraries(osal PUBLIC memutils osal_impl)

# ...

set(CHOSEN_IMPLEMENTATION gcc)   # some logic
add_library(osal_impl INTERFACE)
target_link_libraries(osal_impl INTERFACE ${CHOSEN_IMPLEMENTATION}_impl)

Then the implementation is abstract and cyclic dependency causes linking twice.

The other solution is to be explicit when linking the final stage and repeat one or the other twice:

target_link_libraries(tests PUBLIC additionslib
  system gcc_impl system)

The other solution that you might be interested in on an embedded platform, is optimizations and using OBJECT libraries. That means that you have to list all dependencies on the linking stage, like so:

add_library(osal OBJECT osal/mutex.hpp osal/task.hpp osal/TaskManager.hpp osal/TaskManager.cpp) 
target_include_directories(osal PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/osal)
target_link_libraries(osal PUBLIC memutils)

add_library(system OBJECT sys/system.hpp sys/system.cpp)
target_include_directories(system PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/sys)
target_link_libraries(system PUBLIC osal)

set(platformlib osal system memutils)

add_library(additionslib OBJECT add/additions.hpp add/additions.cpp )
target_include_directories(additionslib PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/add)
target_link_libraries(additionslib PUBLIC ${platformlib})

add_library(gcc_impl OBJECT impls/MutexImpl.cpp impls/TaskImpl.cpp)
target_link_libraries(gcc_impl PUBLIC ${platformlib})

add_executable(tests test.hpp test.cpp main.cpp )
target_link_libraries(tests PUBLIC ${platformlib} additionslib gcc_impl)
KamilCuk
  • 120,984
  • 8
  • 59
  • 111
  • 1
    Your second "explicit linking" does it! I did not think of it this way, since this would mean the "internal" targets of the platform must be used explicitly and anyone who want's to use it should normally not know a lot about the "internal" targets. But this also helps since I do not need to modify the already existing targets. In the end, that means if a link of a target points out that a "internal" target source file has an undefined reference, we need to additionaly link it explicitly somewhere before. Thank you really much! – NetoBF Aug 25 '23 at 11:51
1

system depends on gcc_impl so you need to add:

target_link_libraries(system PUBLIC gcc_impl)

https://godbolt.org/z/Y89h44T3T

Alan Birtles
  • 32,622
  • 4
  • 31
  • 60
  • How would you have targets with multiple implementations? – KamilCuk Aug 25 '23 at 11:18
  • @KamilCuk not sure what you mean? – Alan Birtles Aug 25 '23 at 11:19
  • the problem here is, it's not always gcc_impl. there is also e.g. an impl for other processors and OS'es and so on. So inside system I cannot hardlink against any "outside" occuring impls. – NetoBF Aug 25 '23 at 11:41
  • @NetoBF Just wrap a `if (xxx)` around the `target_link_libraries` statement? Or put the name of the implementation in a variable – Alan Birtles Aug 25 '23 at 11:57
  • that's possible but that means for any new implementation we need to touch the platform again, make a new commit and even a new version even if no source code changed. PIMPL already guarantees a stable ABI and in your versioning-system with multiple submodules and so on you may prefer to touch as less as possible. Also thought about the variable but that means any module that uses the impl needs to have this parameter which makes the CMake more complex. The other solution here is better suited. Nothing needs to be touched inside. Anyway, thanks! – NetoBF Aug 25 '23 at 12:07