3

I want to test that an app is both exiting with a non-zero code AND outputting a particular string. I set the properties like this:

set_tests_properties(
    myapp-test-foobar
PROPERTIES
    WILL_FAIL TRUE
    FAIL_REGULAR_EXPRESSION "^Usage: myapp"
)

But the test passes even if the app's exit code is 0. Although WILL_FAIL is necessary, otherwise the test will fail when the FRE is matched, the exit code is ignored. I searched the cmake docs for an answer to no avail.

EDIT: It turns out that ctest (I'm using v3.19.2) does not check the exit code if either PASS_REGULAR_EXPRESSION or FAIL_REGULAR_EXPRESSION is set. The pass or fail condition is met solely by the RE matching.

A nasty bug has escaped into the wild in one of my apps because of this.

EDIT: The answer suggested in comments below is for handling an app that aborts, i.e. crashes, on the error condition being tested and so is different from my case. Please see my comment below for further reasoning and solution.

msc
  • 1,549
  • 2
  • 12
  • 19
  • 1
    This is a known limitation with CTest... this might help you make progress: https://stackoverflow.com/a/33694733/2137996 – Alex Reinking Feb 06 '22 at 08:31
  • Does this answer your question? [How can I use cmake to test processes that are expected to fail with an exception? (e.g., failures due to clang's address sanitizer)](https://stackoverflow.com/questions/33693486/how-can-i-use-cmake-to-test-processes-that-are-expected-to-fail-with-an-exceptio) – KamilCuk Feb 06 '22 at 11:42
  • Both previous comments point to the same answer which suggest wrapping the app under test in a script or program. While I am sure it would work, it is overly complex for the issue at hand especially as there may be cross-platform issues to resolve in such a script. I have chosen instead to duplicate the affected tests, once with just `WILL_FAIL TRUE` specified and once with that and a FAIL_REGULAR_EXPRESSION specified. Since the tests are targeting error paths, they don't take long to run. – msc Feb 08 '22 at 07:44
  • I would describe the behaviour not as a limitation but as a bug. – msc Feb 08 '22 at 07:55
  • I hit the same issue just now, and ended up doing the same thing, duplicating the test (which is fine for quick tests). – Jack Culhane Jun 17 '22 at 12:01

1 Answers1

2

The fix I described in my comment above does not work. As I recently discovered a mismatched FAIL_REGULAR_EXPRESSION will not cause the test to fail. See this question.

This is because when FAIL_REGULAR_EXPRESSION and WILL_FAIL are set ctest does

if (match FAIL_REGULAR_EXPRESSION || exit_code != 0)

For clarity here are the other cases. When only WILL_FAIL is set, ctest does

if (exit_code != 0)

When PASS_REGULAR_EXPRESSION is set, ctest does

if (match PASS_REGULAR_EXPRESSION)

and when none of the 3 are set, ctest does

if (exit_code == 0)

You can test for both the correct error code and error message by abusing PASS_REGULAR_EXPRESSION as in the following

add_test( NAME myapp-test-ktx2-in-exit-code
    COMMAND myapp -o foo infile.ktx2
    WORKING_DIRECTORY ${MY_WORK_DIR}
)
set_tests_properties(
    myapp-test-ktx2-in-exit-code
PROPERTIES
    WILL_FAIL TRUE
)
add_test( NAME myapp-test-ktx2-in
    COMMAND myapp -o foo input.ktx2
    WORKING_DIRECTORY ${MY_WORK_DIR}
)
set_tests_properties(
    myapp-test-ktx2-in
PROPERTIES
    PASS_REGULAR_EXPRESSION ".* is not a KTX v1 file."
)

If both tests pass we are guaranteed that we have a non-zero exit code and the error message matches PASS_REGULAR_EXPRESSION.

If you want to test both for the correct PASS_REGULAR_EXPRESSION and exit_code == 0 you will similarly need two tests: one to test PASS_REGULAR_EXPRESSION and one without it set.

msc
  • 1,549
  • 2
  • 12
  • 19
  • 1
    "When it is not set, ctest does `match PASS_REGULAR_EXPRESSION || exit code == 0`" - Really? According to [documentation](https://cmake.org/cmake/help/latest/prop_test/PASS_REGULAR_EXPRESSION.html), when PASS_REGULAR_EXPRESSION is set the exit code is completely **ignored**. – Tsyvarev Jul 01 '22 at 10:46
  • The wording of the FAIL_REGULAR_EXPRESSION [documentation](https://cmake.org/cmake/help/latest/prop_test/FAIL_REGULAR_EXPRESSION.html#prop_test:FAIL_REGULAR_EXPRESSION) is identical (except for s/pass/fail/) and in that case ctest is demonstrably checking the exit code when the output fails to match FAIL_REGULAR_EXPRESSION. See [my other question](https://stackoverflow.com/questions/72812239/why-is-ctests-passing-this-fail-test-even-when-fail-regular-expression-doesnt/). – msc Jul 02 '22 at 00:58
  • Read carefully: the wording for [FAIL_REGULAR_EXPRESSION](https://cmake.org/cmake/help/latest/prop_test/FAIL_REGULAR_EXPRESSION.html) and [PASS_REGULAR_EXPRESSION](https://cmake.org/cmake/help/latest/prop_test/PASS_REGULAR_EXPRESSION.html) is **different**. In the first case, the exit code is ignored **only if** the output matches to `FAIL_REGULAR_EXPRESSION`. In the second case the exit code is ignored **in any case**. I just checked: When `PASS_REGULAR_EXPRESSION` is set to "ok", the test `echo invalid` **fails**, despite it exits with zero code. – Tsyvarev Jul 02 '22 at 08:52
  • "regardless of the exit code" vs. "exit code is ignored". As I said in [my other question](https://stackoverflow.com/questions/72812239/why-is-ctests-passing-this-fail-test-even-when-fail-regular-expression-doesnt/) the documentation is not clear. Thanks for clarifying. However, this difference does not change the solution I've given. – msc Jul 03 '22 at 02:22
  • The difference doesn't change the solution you have given, but it conflicts with your description (`match PASS_REGULAR_EXPRESSION || exit code == 0`). Please, reword this description for do not confuse future readers. – Tsyvarev Jul 03 '22 at 08:30
  • I have done it. – msc Jul 04 '22 at 09:13
  • You have two snippets with pseudo code. According to **outer** description, they differs only in `WILL_FAIL`. But **different number of branches** in your snippets confuses a lot: changing WILL_FAIL should only **invert results** in branches. Actually, your first snippet describes the case "WILL_FAIL is True and FAIL_REGULAR_EXPRESSION is set", and the second snippet describes the case ""WILL_FAIL is False and FAIL_REGULAR_EXPRESSION is not set". These conditions are not quite *symmetric*. Probably, you want the second case to describe "WILL_FAIL is False and PASS_REGULAR_EXPRESSION is set". – Tsyvarev Jul 04 '22 at 10:50
  • @Tsyvarev, I do not understand what it is you refer to as the "outer description" nor which are the "snippets". Are the latter the pseudo code for the ctest behavior or the tests I proposed? Please clarify. If you are referring to the description of ctest behavior, please propose a clearer alternative. – msc Jul 06 '22 at 01:24
  • Yes, I mean the pseudo code where you encode the ctest behavior. Because you have two chunks of that pseudo code, I refer to them as "snippets". You write "This is because ctest does <..> when WILL_FAIL is set. When it is not set, ctest does <..>", according to which one could assume that snippets differs **only** in `WILL_FAIL` setting. But actually the second snippet describes **wider conditions** than the first one. – Tsyvarev Jul 06 '22 at 07:17
  • It seems that 3 *one-line* snippets would better suited for your answer: 1. `match FAIL_REGULAR_EXPRESSION || exit code != 0` (when FAIL_REGULAR_EXPRESSION and WILL_FAIL are set) 2. `match PASS_REGULAR_EXPRESSION` (when PASS_REGULAR_EXPRESSION is set and WILL_FAIL is not set). 3. `exit code != 0` (when only WILL_FAIL is set). The first one corresponds to your code in the question post, so it should clearly show why that code doesn't work as intended. The second and third ones correspond to your "real" code in the answer. – Tsyvarev Jul 06 '22 at 07:23
  • What about `exit_code == 0` when WILL_FAIL and PASS_REGULAR_EXPRESSION is not set? – msc Jul 08 '22 at 09:34
  • But the verdict `exit_code == 0` is never needed for your tests, where you verify failing the program in case of incorrect input. But of course, no one prevents you from describing that case. – Tsyvarev Jul 08 '22 at 10:03