0

I have a test project written on C# with thousands of tests. Now I'm updating NUnit related stuff to the latest versions. After I did that, a couple tests (5 out of 2500) started failing. An interesting thing is that when I run them from Visual Studio - they pass. They fail only when running from nunit-console. They not simply fail because of some unhandled exception, but the assertion doesn't pass (expected differs from actual).

Here are versions I update to:

  • NUnit-Console - 3.12.0
  • NUnit Framework - 3.13.1
  • NUnitTestAdapter - 3.17.0

Here are current versions that work fine:

  • NUnit-Console - 3.10.0
  • NUnit Framework - 3.5.0
  • NUnitTestAdapter - 3.15.1

I'm very confused, can't see any reason why this is happening. Maybe someone experienced the same and knows at least a workaround?

Siarhei Machel
  • 103
  • 8
  • Please provide a minimal example test here with the relevant assertion condition! How are your test runners defined in detail (parallel execution?) – Secundi Feb 10 '21 at 11:32
  • The problem is that I cannot replicate it with a minimal example (otherwise I'd find the root cause and fix it). The production code has tons of dependencies, cannot bring it in unfort. I don't think there is a problem in my code (otherwise it wouldn't work on other NUnit versions) – Siarhei Machel Feb 10 '21 at 12:11
  • But can you provide the general scheme of your assertion conditions? Could you deduce, what might be special about those five ones failing now in comparison to the other ones? From the current information solely, I doubt anyone will be abe to help you here (except their are known bugs for the newer NUnit-versions I'm not aware of here). – Secundi Feb 10 '21 at 12:22
  • It looks like it is a concurrency issue: when I run only TestFixture that the failing tests belong to (by assigning a special category) - they do pass even on nunit-console. On the project level though TestFixtures run in parallel. Please do not close the question for a while. – Siarhei Machel Feb 10 '21 at 12:36
  • Indeed, it is a concurrency issue: someone replaces a static variable in different TestFixtures (that may run in parallel) thus changing the behavior. (facepalm). Seems like concurrency model has slightly changed in newer versions and this revealed the problem. The question can be closed now. – Siarhei Machel Feb 10 '21 at 13:45

0 Answers0