ctest - run tests having 2 labels

Question

lest assume I have some project that consists of N domains (D1, D2, ... , DN) . Each domain has tests of two kinds: lest say UT and MT. They are defined like so:

add_test(
  NAME Di_UT
  COMMAND <blah>
)

add_test(
  NAME Di_MT
  COMMAND <blah>
)

And I'd like to be able to filter them by labels. So I add the labels:

set_tests_properties(Di_UT PROPERTIES LABELS "UT;Di")
set_tests_properties(Di_MT PROPERTIES LABELS "MT;Di")

Then I execute ctest:

ctest -L Di

will execute all tests for domain Di, and of course the opposite:

ctest -L UT

Will execute all tests with UT label.

But how to filter by both labels? execute only UT for domain Di?

From what i observe, passing multiple -L causes them to overwrite. (the last one has effect). Any other ideas? my cmake version is

ctest version 3.13.4

This was filed upstream as a feature request in https://gitlab.kitware.com/cmake/cmake/-/issues/21087 and is beint worked on in MR https://gitlab.kitware.com/cmake/cmake/-/merge_requests/5329 so **hopefully** with CMake 3.21 the answers below will be obsolete. — Adriaan de Groot, Feb 27 '21 at 21:43

score 2 · Accepted Answer · answered Oct 21 '19 at 11:47

It seems that such thing is simply not supported. So as a workaround, I am simply adding a 3rd label containig the other two - and then pas it to ctest. So:

add_test(
  NAME Di_UT
  COMMAND <blah>
)

set_tests_properties(Di_UT PROPERTIES LABELS "UT;Di;UT-Di")

add_test(
  NAME Di_MT
  COMMAND <blah>
)
set_tests_properties(Di_MT PROPERTIES LABELS "MT;Di;MT-Di")

So then i can execute like so:

execute all UT:

ctest -L UT

execute all Tests for domain Di:

ctest -L Di

execute only UT for Di:

ctest -L Ut-Di

This seems only available option.

score 2 · Answer 2 · answered Oct 01 '21 at 07:26

2

The possibility to use regexes in -L has been added in (at least) v3.21, see here

answered Oct 01 '21 at 07:26

capitalaslash

240
4
10

nice :) will remember about this next time – murison Oct 01 '21 at 09:52
1

The `-L` and `-LE` arguments always did regex matching, but v3.21 finally documented this and also changed the behavior when given multiple `-L` or `-LE` arguments – Cody Martin Aug 27 '22 at 20:33

score 1 · Answer 3 · answered Mar 20 '20 at 17:27

I've been struggling with this too. It's such a pity that such feature is not supported, especially if you consider that the logical or works:

ctest -L 'label1|label2'

What I came up with is to create my own bash function which boils down to the following:

ctest -R "$(echo "$(ctest -N -L label1 | awk '/Test #/ { print $3}')" | tr '\n' '|')" -L label2

I know, it's a bit ugly, but let's break it down.

You ask ctest to give you all the tests with label1 (-N, so you only list them).
Then use awk to extract the test name (without the string Test #xyz: first).
Then you put all those lines into a single regexp, by replacing \n with |.
You use this string as a regexp for ctest, combined with a request for a new label. This makes ctest run tests which have label2 and match any of the test names you found at step 1.

It's quite cumbersome, especially if you want to run tests that have N labels.

Cody Martin · Answer 4 · 2022-08-27T20:37:57.250

Based on local testing, with CMake 3.18.2, multiple labels are not respected, it just listens to the final one

ctest -L Di -L UT <--- equivalent to ctest -L UT

This is consistent with your findings for CMake 3.13.4. The logical | operator is supported because the labels use regex matching under the hood:

ctest -L "Di|UT" <-- all tests that have either label

Unfortunately, the logical & operator is not.

In CMake 3.21, they added documentation of label matching behavior in this issue and merge request. Testing with CMake+CTest 3.21.3 indicates the default behavior for multiple label arguments changed:

ctest -L Di -L UT <--- only tests that have both labels

which matches the new documentation

Using more than one -L option means "match all of these".

tl;dr: if possible, upgrade to CMake 3.21+

ctest - run tests having 2 labels

4 Answers4