How should I setup script-based source generation with CMake?

Question

I'm managing the build of a certain C++ repository using CMake. In this repository, and among other things, I have a bunch of .hpp files in a directory, and for each of these, I need to compile a generated bit of source code, which depends on its contents.

Naively, I would do this by generating a correspond .cpp file, and include that file in the source files of the CMake target for the library or executable I'm building. Now, since I don't actually need the source files themselves, I could theoretically just arrange for the compiler to get its source from the command-line instead.

My question: How would I set up this source generation and compilation, using CMake as idiomatically as possible?

Notes:

Assume I have, say, a bash script which can read the .hpp and generate the .cpp on the standard output.
CMake version 3.24 or whichever you like.
Should work on Unix-like operating systems, and hopefully on Windows-like OSes other than the fact that the bash script will fail.
Please comment if additional information is necessary to answer my question.

This is just a classic use of `add_custom_command`. Did you try that already / is something not working? — Alex Reinking, Aug 07 '22 at 23:48
@AlexReinking: I was assuming whether there was something higher-level than add_custom_command; and whether I should actually generate files or just generate sources for the compiler to eat. — einpoklum, Aug 08 '22 at 19:14

Alex Reinking · Answer 1 · 2022-08-08T12:39:20.907

Let's assume for the sake of portability that you have a Python script, rather than a bash script that manages your code generation. Let's say that it takes two arguments: the source .hpp file and the destination .cpp file. We'll assume it is in your source tree under ./tools/codegen.py.

Now let's assume that your .hpp files are in ./src/genmod for "generated module" because the sources for these headers are generated by codegen.py.

Finally, we'll assume there's a final executable target, app, with a single source file, ./src/main.cpp.

Here's a minimal build that will work for this, with some step-by-step discussion.

We start with some boring boilerplate.

cmake_minimum_required(VERSION 3.24)
project(example)

This probably works on earlier versions, I just haven't tested it, so YMMV. Now we'll create the executable target and link it to our generated sources preemptively. Note that the dependent target does not need to exist before calling target_link_libraries.

add_executable(app src/main.cpp)
target_link_libraries(app PRIVATE genmod)

Now we'll find a Python interpreter and write down an absolute path to the codegen tool.

find_package(Python3 REQUIRED)
set(codegen_py "${CMAKE_CURRENT_SOURCE_DIR}/tools/codegen.py")

Next we'll construct the list of input headers. I'm imagining there are three: A.hpp, B.hpp, and C.hpp.

set(input_headers A.hpp B.hpp C.hpp)
list(TRANSFORM input_headers PREPEND src/genmod/)

I used list(TRANSFORM) here to save some typing. Now we'll just create an object library called genmod, which will "hold" the objects for the generated C++ files.

add_library(genmod OBJECT)

And now comes the real meat. For each of the headers, ...

foreach (header IN LISTS input_headers)

we'll construct absolute paths to the header and generated source files ...

    string(REGEX REPLACE "\\.hpp$" ".cpp" gensrc "${header}")
    set(header "${CMAKE_CURRENT_SOURCE_DIR}/${header}")
    set(gensrc "${CMAKE_CURRENT_BINARY_DIR}/${gensrc}")

and then write a custom command that knows how to call codegen.py. We specify the outputs, command arguments, and dependencies. Don't forget to include the generator script as a dependency, and never forget to pass VERBATIM to ensure consistent, cross-platform, argument quoting.

    add_custom_command(
        OUTPUT "${gensrc}"
        COMMAND Python3::Interpreter "${codegen_py}" "${header}" "${gensrc}"
        DEPENDS "${header}" "${codegen_py}"
        VERBATIM
    )

Finally, we attach this source to genmod.

    target_sources(genmod PRIVATE "${gensrc}")
endforeach ()

We can test this build using Ninja's dry-run feature to make sure the commands look correct.

$ cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=Release
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /home/reinking/.venv/default/bin/python3.9 (found version "3.9.2") found components: Interpreter 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/reinking/test/build

$ cmake --build build -- -nv
[1/8] cd /home/reinking/test/build && /home/reinking/.venv/default/bin/python3.9 /home/reinking/test/tools/codegen.py /home/reinking/test/src/genmod/A.hpp /home/reinking/test/build/src/genmod/A.cpp
[2/8] cd /home/reinking/test/build && /home/reinking/.venv/default/bin/python3.9 /home/reinking/test/tools/codegen.py /home/reinking/test/src/genmod/B.hpp /home/reinking/test/build/src/genmod/B.cpp
[3/8] cd /home/reinking/test/build && /home/reinking/.venv/default/bin/python3.9 /home/reinking/test/tools/codegen.py /home/reinking/test/src/genmod/C.hpp /home/reinking/test/build/src/genmod/C.cpp
[4/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/genmod.dir/src/genmod/A.cpp.o -MF CMakeFiles/genmod.dir/src/genmod/A.cpp.o.d -o CMakeFiles/genmod.dir/src/genmod/A.cpp.o -c /home/reinking/test/build/src/genmod/A.cpp
[5/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/genmod.dir/src/genmod/B.cpp.o -MF CMakeFiles/genmod.dir/src/genmod/B.cpp.o.d -o CMakeFiles/genmod.dir/src/genmod/B.cpp.o -c /home/reinking/test/build/src/genmod/B.cpp
[6/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/genmod.dir/src/genmod/C.cpp.o -MF CMakeFiles/genmod.dir/src/genmod/C.cpp.o.d -o CMakeFiles/genmod.dir/src/genmod/C.cpp.o -c /home/reinking/test/build/src/genmod/C.cpp
[7/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/app.dir/src/main.cpp.o -MF CMakeFiles/app.dir/src/main.cpp.o.d -o CMakeFiles/app.dir/src/main.cpp.o -c /home/reinking/test/src/main.cpp
[8/8] : && /usr/bin/c++ -O3 -DNDEBUG  CMakeFiles/genmod.dir/src/genmod/A.cpp.o CMakeFiles/genmod.dir/src/genmod/B.cpp.o CMakeFiles/genmod.dir/src/genmod/C.cpp.o CMakeFiles/app.dir/src/main.cpp.o -o app   && :

And indeed we can see that the commands are what we'd naturally expect them to be.

Won't this generate the sources only at build time? Rather than config time? — einpoklum, Aug 08 '22 at 06:45
Where in your question did you specify that you wanted the source files generated at config time? In any case, I wouldn't do that... incrementally building just one file would force the regeneration of all of them, not to mention all the unrelated work. And doing it at configure time locks you out of parallelism. — Alex Reinking, Aug 08 '22 at 11:59
I didn't... which is why I gave an upvote. But - generation at config time could also in theory only happen if the file is missing. If in no other way, then by checking for the sources' existence. Anyway, I eventually went the Python route with a bash script as fallback if Python is missing. Perhaps you should add one should make it a python2-and-3 compatible script for maximum portability. — einpoklum, Aug 08 '22 at 12:21
Sure you could check timestamps and whatnot at configure time, but you can't avoid all the other work your configure step might do, plus the generated build system would do that work for you anyway. Let each part do what it's good at. With Python 2 [well past end of life](https://www.python.org/doc/sunset-python-2/), it is no longer worthy of consideration. — Alex Reinking, Aug 08 '22 at 12:35
Python 2 was [still the default in Ubuntu in 2019](https://askubuntu.com/questions/1165360/why-is-python-2-7-still-the-default-python-version-in-ubuntu). So, maybe in 10 years or so it would be unworthy of consideration. — einpoklum, Aug 08 '22 at 14:58
@einpoklum - the answer to that question directly contradicts you: "Python 2 isn't installed by default in 18.04 and versions released after that." Given that 18.04 is the oldest supported LTS, I have no idea where you're getting 10 years from. — Alex Reinking, Aug 08 '22 at 16:06
18.04 was the LTS version of Ubuntu when that question was asked. An installation in 2019 cannot be disregarded for a good number of years, say 10-15. So, 10 years from now we start assuming python3 is the default version. — einpoklum, Aug 08 '22 at 17:41
I think that's an argument - at best - for keeping the `3` in the `#!/usr/bin/env python3` she-bang. I would be astonished if older systems had no way of installing some version of Python 3. In general, I don't think it's wise to support EOL software in new systems. Python 2 no longer receives security updates. I personally feel ethically uncomfortable indulging users who refuse to move off of vulnerable software. — Alex Reinking, Aug 08 '22 at 17:47
I guess that's fair enough, but - Python::Interpreter will run the distro's default interpret, wouldn't it? — einpoklum, Aug 08 '22 at 19:13
I wrote `find_package(Python3 REQUIRED)` and used the target `Python3::Interpreter`, so it will use whichever interpreter it finds that is some version of Python 3, not 2. Maybe that's a system interpreter, maybe it's a conda environment or venv. — Alex Reinking, Aug 08 '22 at 19:18
[This](https://github.com/eyalroz/gpu-kernel-runner/blob/main/CMakeLists.txt) is what I'm doing: (and the [scripts](https://github.com/eyalroz/gpu-kernel-runner/tree/main/scripts)). — einpoklum, Aug 08 '22 at 19:26

How should I setup script-based source generation with CMake?

1 Answers1