CMake shared project subdirectories avoid rebuilding

Question

For a project, I am using Android gradle scripts with CMake, gradle plugin is version 3:0:0, CMake version 3.6. Both gradle and CMake files are pretty simple and uninteresting (just defining the files used - I can still copy-paste them as required).

I have the following project structure; basically a codebase producing a few tens of .so files (the native part for the Android packages that get packaged into an apk, thereby called 'Executables'), which all depend on the same shared library code (static libraries, thereby called 'Libraries'). The Library code is still (relatively) volatile, so I wish the Executables to have project-level dependencies on them, so that whenever the Executables are built, the Libraries are rebuilt on-demand every time their code is changed. The structure looks like:

+ LibProjects/
---Bin/ (Originally empty)
---Lib1/CMakeLists.txt (+sources files, same level as the CMakeLists.txt)
...
---Lib10/CMakeLists.txt (same)
+ Executables/
---Executable1/CMakeLists.txt (source files here)
--------------/AndroidFiles/build.gradle (and other android project files)(points to the CMakeLists.txt)
...
---Executable40/CMakeLists.txt

The Libraries' CMakeLists redirect their output into the Bin folder using

set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY {CMAKE_CURRENT_SOURCE_DIR}/../Bin/${ANDROID_ABI}/${LibraryName})

The executable's projects add dependencies on the libraries "as normal"

add_subdirectory(${PROJECTS_ROOT}/LibProjects/${LibraryName} ${PROJECTS_ROOT}/Framework/Bin/Android/${ANDROID_ABI}/${LibraryName})...

Everything almost works, in the sense that I can get sensible executables and the Executables trigger builds of the libraries.

The problem is that when building the executables sequentially, each one does NOT reuse the library project outputs of the other ones: When I build Executable1, it will build all libraries (normal) and then it will build itself. Afterwards, when I build Executable2, it will NOT reuse the libraries that were already built for Executable1, and so on - this effectively increases my build time by a factor of ~10.

I can find the output of the build of each library inside the /Bin folder as expected, but they are not reused across executables - there are no CMake "project files" (is this the correct term) in the bin folder, all of them get generated inside the executable build directory.

The problem I am trying to resolve is the build times stemming from the fact that each library gets rebuilt for each executable.

At the moment the solutions I am considering is to somehow instruct CMake to use the Bin folder (or another folder) as a working folder for each library in its own folder instead of with the executable, hoping that the gradle android plugin will be smart enough to then spot that neither the cmakefiles nor the object files need to be regenerated, and avoid the rebuild.

The restriction that I have is that I cannot restructure the codebase itself, and that each Executable must be buildable separately of the others - there is absolutely no possibility of a top-level CMake - each Executable should be able to be triggered on its own.

Answer 1

CMake can guess if the build is up-to-date by reading informations froms the current build directory.

When you run CMake manualy in Executables/<x> directory, cmake retrieve information from the build directory associated to Executable/<x> directory. It then check if the timestamp of the built file correspond to the last build performed in this build directory. If not, it rebuild. What happen is that: Lib1 library file is built after you build Executable1, then you run cmake in Executalbe2, it compares the timestamp of Lib1 target file, see that this file was not produced by this instance of the cmake build and then rebuild the lib. And so on.

So you have two options:

1- Either you build the library and install their target files in the bindirectory (using install cmake command and make install bash command for exemple). Then in the Executalbe<x>/CMakeLists you use find_library command instead of add_subdirectory.

2- Or you create a super project which has the following structure:

+  supper_project
---CMakeLists.txt #add_subdirectory(LibProjects/lib<x>)... add_subdirectory(Executables/Executalbe<x>)...
  + LibProjects/
  ---Bin/ (Originally empty)
  ---Lib1/CMakeLists.txt (+sources files, same level as the CMakeLists.txt)
  ...
  ---Lib10/CMakeLists.txt (same)
  + Executables/
  ---Executable1/CMakeLists.txt (source files here)
  --------------/AndroidFiles/build.gradle (and other android project files)
  (not any more:points to the CMakeLists.txt)
  ...
  ---Executable40/CMakeLists.txt

Answer 2

I managed to work around this problem - but in the end it was by working around rather than with CMake.

I removed the CMakeFile-level dependencies (add_subdirectory) and only left the libraries at the linking level (target_link_libraries Executable [the library files])

Afterwards, I created gradle scripts for each library and added dependencies to these scripts in each application gradle script, so that the building of the libraries gets triggered by gradle dependencies instead of CMake dependencies. It's slower than it would be if gradle could be avoided, but much faster than rebuilding every time, and the overhead is at least constant (a few seconds per project).

Answer 3

I think problem lies in the way you have defined your dependencies.

For each executable you are creating separate targets using add_subdirectory. e.g. for executable 1 you have add_subdirectory(${PROJECTS_ROOT}/LibProjects/${Library1}) and for executable 2 also you have add_subdirectory(${PROJECTS_ROOT}/LibProjects/${Library1}), so cmake will create two separate targets for same library1 in each of the executable's subdirectory and thus it will create separate timestamp and cache files. That is why it looks that it is building the same library for multiple times, but in fact for cmake they are different targets.

To fix this you can include all libraries in top level CMakeLists.txt using add_subdirectory and them in each executable's CMakeLists.txt add the dependency using add_dependencies command.