2

I have a Boost.Test based testsuite, where usually a number of tests is combined into one binary.

If I run them separately from CTest, I get an XML file with one entry per binary, which is not detailed enough in the CI output. There is a nice script that has CTest interrogate the test binaries and run the tests separately, which gives good results but also increases test time heavily.

Is there a "mixed mode" approach where I can tell CTest to run multiple tests that exist in the same binary together, and only run subsets when the combined test reports a failure?

For example, the output generated from CTest has nine entries, one for each binary registered with add_test(). Each binary runs several tests, and the only way to get CTest to report them separately is to instruct it to invoke the binary once for each subtest, which consumes a lot of time.

Simon Richter
  • 28,572
  • 1
  • 42
  • 64
  • Would be good if you could clarify "I get an XML file with one entry per binary, which is not detailed enough in the CI output" – Raffi Apr 22 '20 at 17:25
  • @Raffi, I have nine binaries performing 300 tests. The CTest report has nine entries, unless I instruct CTest to run each binary once for each contained test, which takes ages. – Simon Richter Apr 23 '20 at 06:50
  • Since you are using Boost.Test, you could bypass ctest completely and use Boost.Test to generate a JUnit XML report, which can be picked up by Jenkins. See https://www.boost.org/doc/libs/1_72_0/libs/test/doc/html/boost_test/test_output/log_formats.html – sakra Apr 24 '20 at 18:36
  • @sakra, not entirely -- this fails if one of the tests crashes, and there are other tests that are not Boost.Test based (mostly language integration tests where our code is called from a script). – Simon Richter Apr 26 '20 at 14:00
  • @simon richter: thanks. How about asking Boost.Test to generate the XML output instead of CTest? Boost.Test will produce a machine parseable XML fr each test case. The ctest output is there to track the result of each `add_test` command. – Raffi Apr 29 '20 at 07:39
  • @SimonRichter: `this fails if one of the tests crashes`: well this is the responsibility of Boost.Test and if there is a bug, I would be happy to fix it :) OTOH we cannot handle everything in Boost.Test, this is why `ctest` is valuable. Boost.Test will report unit cases, `ctest` will report `add_test` commands. However keep in mind that a crash is sometimes the consequence of running a specific path that puts the program in a particular state: isolating the execution of each test may prevent you from detecting the root cause of the crash. – Raffi Apr 29 '20 at 07:54

0 Answers0