0

I tried to use Combinatorial/Values attributes in NUnit.

While it works locally, the test decorated with those attributes are skipped in TeamCity build running NUnit 2.6.2—the build report indicates that the test was ignored, without giving any details about the reason to ignore it.

According to the documentation, Combinatorial attribute (as well as the Values attribute) exists in NUnit 2.5, and so I would expect it to still be supported in NUnit 2.6.

Why is the test ignored?

Here's a most basic example which reproduces the issue. Locally, both tests run and give the exact same results. On TeamCity, only Test1 is executed, and Test2 is marked as ignored.

[TestCase("a", "1")]
[TestCase("a", "2")]
[TestCase("b", "1")]
[TestCase("b", "2")]
public void Test1(string x, string y) {
    Assert.AreEqual(x + "," + y, string.Format("{0},{1}", x, y));
}

[Test, Combinatorial]
public void Test2([Values("a", "b")] string x, [Values("1", "2")] string y) {
    Assert.AreEqual(x + "," + y, string.Format("{0},{1}", x, y));
}
Charlie
  • 12,928
  • 1
  • 27
  • 31
Arseni Mourzenko
  • 50,338
  • 35
  • 112
  • 199

1 Answers1

1

Your question could really do with a code example to make it clear what you are doing...

Nevertheless... I'll give it a go...

CombinatorialAttribute merely tells NUnit how to combine values provided for the individual parameters of the test method in order to create test cases. If you haven't provided such values for each parameter, then there is nothing to combine and no test cases are created.

Individual values are provided by using ValuesAttribute and similar attributes. Combinatorial is actually the default when such values are provided, so it's not actually needed.

My guess, without seeing your code, is that you haven't given CombinatorialAttribute anyvalues to combine.

You may wonder why there is no error message in this case. That's because some people create tests for which there is sometimes data available and sometimes not. Probably, a suppressable warning would be useful here.

Since your problem only shows up in TeamCity (I added a tag) this could simply be a problem with TeamCity. One thing to be aware of is that it is sometimes possible for TeamCity to report something different than what is contained in the NUnit result file... so you should examine that file to figure out whether NUnit is really skipping the tests or if TC merely thinks it is.

Charlie
  • 12,928
  • 1
  • 27
  • 31
  • It is nice to have answers from the maintainer of NUnit in person. Actually, I did use `Values` attribute (that I mistakenly named “Value” in my original question); I edited my question to add an example. Also note that the test runs correctly on my machine, all the checks are done exactly as expected, so I wouldn't expect a error to be in the code I've written. It's only when it runs from TeamCity that the combinatorial tests are marked as ignored. – Arseni Mourzenko May 19 '21 at 18:10
  • I was thinking that the issue may be related to async tests, so I simplified them. The problem exists even with ordinary, non-async tests. I edited my answer once again to provide this simpler example. – Arseni Mourzenko May 20 '21 at 08:03
  • I added a teamcity tag and some further ideas in the answer. – Charlie May 20 '21 at 23:41