0

I added a simple test to ctest with the following lines in a .cmake file:

add_test( NAME ktxsc-test-many-in-one-out
    COMMAND ktxsc -o foo a.ktx2 b.ktx2 c.ktx2
)
set_tests_properties(
    ktxsc-test-many-in-one-out
PROPERTIES
    WILL_FAIL TRUE
    FAIL_REGULAR_EXPRESSION "^Can't use -o when there are multiple infiles."
)

The test passes and the TestLog shows

----------------------------------------------------------
Test Pass Reason:
Error regular expression found in output. Regex=[^Can't use -o when there are multiple infiles.]
"ktxsc-test-many-in-one-out" end time: Jun 30 16:34 JST
"ktxsc-test-many-in-one-out" time elapsed: 00:00:00
----------------------------------------------------------

If I change FAIL_REGULAR_EXPRESSION to

FAIL_REGULAR_EXPRESSION "some rubbish"

the test still passes even though the app is printing the same message as before. This time the test log shows

----------------------------------------------------------
Test Passed.
"ktxsc-test-many-in-one-out" end time: Jun 30 16:53 JST
"ktxsc-test-many-in-one-out" time elapsed: 00:00:00
----------------------------------------------------------

which is what I normally see when no *_REGULAR_EXPRESSION is set.

Why is this happening? How can I get ctest to fail the test when the FAIL_REGULAR_EXPRESSION doesn't match?

msc
  • 1,549
  • 2
  • 12
  • 19
  • You set property `WILL_FAIL`, so failing the test means success. Failing could be determined by one of two aspects: 1. Matching to `FAIL_REGULAR_EXPRESSION`. This is your first case. 2. Non-zero exit code. Probably, it is your second case. Normally, one sets `FAIL_REGULAR_EXPRESSION` property for output, which signals about failing the program, but the program still exits with zero code. – Tsyvarev Jun 30 '22 at 08:10
  • Thanks @Tsyvarev. I'll create an answer based on this. – msc Jul 01 '22 at 09:30

1 Answers1

0

Thanks to @Tsyvarev for this answer.

ctest is doing

match FAIL_REGULAR_EXPRESSION || exit code != 0

to determine if a test has failed. If the match succeeds the exit code is not checked. If the match fails the exit code is checked and the failed match is ignored.

The documentation is unclear. I've read it many times and still didn't figure out this behavior until steered in the right direction by @Tsyvarev. I also find this a poor behavioral choice. All my tools output both a non-zero error code and a message when there is an error condition. I need to test both. I previously asked this question about that. It requires duplicating tests.

msc
  • 1,549
  • 2
  • 12
  • 19