How to generate source files during the build time using python script in CMake and build them successfully

Question

I have a requirement where I need to auto generate the list of source files from a python script during the build time and build them. Please note that the file names are generated within the python script and the list is huge. Here is the: directory structure

ROOT
|
-- build
   |
   -- CMakeLists.txt
   __ my_python_script.py
|
-- sub_folder
      |
      -- build
           |
           -- CMakeLists.txt - This Cmake file will glob the sources and run target_source command to generate the library
      -- auto_gen_dir
              |
              -- Script will generate the source/header files here

I tried using add_custom_command and add_custom_target but both are generating the source files but are not compiling them. Due to this the build fails with undefined reference errors. When I run the build command second time then it goes through the unbuilt files [auto generated ones] and build it successfully. This is always a problem with a clean/first time build.

Just FYI. The other method that I tried is to auto generate the source files during the configuration time using execute_process() command and it generates the source files and are compiled in the build time. This has no issues.

This is the code snippet that I tried in my CMake from ROOT/build/CMakeLists.txt:

add_custom_command(TARGET my_target
   POST_BUILD #Please note I have used PRE_LINK too. PRE_BUILD will work with only Visual Studio geneartor
   COMMENT "Auto generating source/header files for 'amdb_core' library"
   COMMAND ${CMAKE_COMMAND} -E env
   ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
   COMMENT "Auto generating source/headers is complete"
   VERBATIM)

"Please note that the file names are generated within the python script and the list is huge." - Still, the only way is to specify all generated files in OUTPUT parameter of `add_custom_command`, and list all these files in `add_executable`/`add_library` for compile them. — Tsyvarev, Jul 21 '23 at 00:51

score 0 · Answer 1 · answered Jul 21 '23 at 19:32

To expand on @Tsyvarev's comment, CMake needs to know your source list before build-time, which is why it worked when you generated your sources at configure-time. Now, that doesn't mean your sources have to exist at build-time. Try modifying your generator script to have two modes of operation. One outputs a list of files it would generate, the other generates those files. Then run your new list-mode at configure-time to populate your target's source list, and set generator-mode to be a dependency of your target so its sources are created during build-time.

KamilCuk · Answer 2 · 2023-07-24T19:11:02.947

So generate the files:

add_custom_command(
   OUTPUT ${AUTO_GEN_DIR}/file1.c ${AUTO_GEN_DIR}/file2.c ${AUTO_GEN_DIR}/file3.c etc..
   COMMAND ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
)

And then compile them:

add_executable(your_executable ${AUTO_GEN_DIR}/file1.c ${AUTO_GEN_DIR}/file2.c ${AUTO_GEN_DIR}/file3.c etc..)

Typically, you would generate everything inside CMAKE_CURRENT_BINARY_DIR. Generating them in source is confusing.

not to list the actual file names

You can execute the python script for cmake beforehand to get only filenames for example:

execute_process(
   COMMAND ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py GET_LIST_OF_FILES
   RESULT_VARIABLE FILES
)
add_custom_command(
   OUTPUT ${FILES}
   COMMAND ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
)

You could just execute the python script from cmake on configuration stage with execute_process and then after it just use glob.

And finally, you can wrap your cmake with top-level build tool, like make.

Thanks KamilCuk. I was just thinking not to list the actual file names because in future if we add more files to be generated then the update to this CMake file can be skipped. — Sriram, Jul 24 '23 at 16:50

How to generate source files during the build time using python script in CMake and build them successfully

2 Answers2