1

in my team we develop a library which coordinates multiple applications over some network back-end. The building is handled by CMake 3.10.4 and we run our tests with CTest. We currently support only Linux, but Windows and MacOS support is on the horizon.

We also provide example applications which we configure and build as tests in CTests.

How can we:

  1. run 2 example programs with CTest as a single test in parallel?
  2. keep the implementation as system-independent as possible?
  3. capture both STDOUT separately (interleaved is fine too)?

What we tried

We created a CMake script which is called by CTest. It executes the following using execute_process:

# DUMMY_A, DUMMY_B are the dummy application
execute_process(
  COMMAND ${DUMMY_A} ${DUMMY_CONFIG} SolverOne MeshOne
  COMMAND ${DUMMY_B} ${DUMMY_CONFIG} SolverTwo MeshTwo
  WORKING_DIRECTORY ${DUMMY_RUN_DIR}
  RESULTS_VARIABLE DUMMY_RESULTS
)

This does not work reliably as DUMMY_B may return prior to DUMMY_A. In this case, DUMMY_A still writes to STDOUT which leads to a SIGPIPE (broken pipe) and thus a failing test.

Further options we considered:

  • Using a shellscript to run both programs in parallel, which is system dependent.
  • Using a python script to run one program as subprocess, which obviously requires python.
  • Allowing the tests to fail with SIGPIPE which we can check via RESULTS_VARIABLE. This could hide actual failing tests though.
Frederic Simonis
  • 21
  • 4
  • @Tsyvarev there are two very similar options for `execute_process`: `RESULT_VARIABLE` for the exit code of the last process and `RESULTS_VARIABLE` (note the `s`) for a list of all exit codes. `execute_process` already runs the processes in parallel, the question is how to prevent the broken piping. – Frederic Simonis Apr 03 '20 at 10:28
  • Oh, didn't notice terminating `S` in the option's name, CMake 3.10+ actually has this option. So, would you don't bother about output of `DUMMY_A`, you may wrap its execution with additional `execute_process` with `OUTPUT_QUIET` option, so it no longer writes into the pipe and cannot get SIGPIPE. But if the application's output is significant, then you second COMMAND should take to things in parallel: 1. output its stdin. 2. Execute `DUMMY_B`. Parallel execution is needed again... Instead of overcoming pipe it seems to be simpler to create 2 or 3 platform-specific small shell wrapper scripts. – Tsyvarev Apr 03 '20 at 11:07
  • @Tsyvarev Thanks the input! Writing a few minimal scripts is definitely easier to maintain than crowbaring the functionality into CMake. Would you mind posting your comment as an answer so I can mark it as the solution? – Frederic Simonis Apr 03 '20 at 13:30
  • Suggestion to turn away from some method without specifying replacement could be anything but "solution" :) As for shell scripting, while I clearly understand the Linux approach for create a background process and wait for it, for Windows it could be quite harder: https://stackoverflow.com/questions/14747963/windows-batch-run-a-process-in-background-and-wait-for-it. Unless you need only a single parallel test, you may still consider scripting in non-shell languages or even C/C++ program. – Tsyvarev Apr 03 '20 at 14:32

0 Answers0