2

Within my freestyle Jenkins-job I´m executing unit-tests via the "execute Windows batch-command"-step:

call "C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" MyAssembly.dll
call SomeOtherProcess

As there are tests that fail, I´d expected the build to fail as well. However the test-publishing-step for NUnit markes the build as unstable:

Build step 'Publish NUnit test result report' changed build result to UNSTABLE

If I´d remove the SomeOtherProcess-line from my batch-script, everything works fine and the errors produced by nunit are reported as error in the build-process.

I read a similar issue for the JUnit-test-reporter (Jenkins JUnit Plugin reports a build as unstable even if test fails). Obviously that reporter does not even support failing the build. I´m not sure if the same applies to the NUnit-reporter as well.

MakePeaceGreatAgain
  • 35,491
  • 6
  • 60
  • 111

1 Answers1

1

The plugin set the result to UNSTABLE because the option, by default, failedTestsFailBuild is set to false.

You can control the behavior applies of NUnit, setting failedTestsFailBuild to true. When you call from a scripted or declarative pipeline.

The issue is the GUI doesn't reflect all the options available for this plugin. There is a PR opened to include this option inside the freestyle pipeline, you can vote up or ask the status of this PR.

To change to an error you need to catch the unstable result and set it to failure using a plugin or a scripted or declarative pipeline.

sumitani
  • 228
  • 2
  • 11
  • Hmmm, that doesn´t explain why omitting the last line from my batch-script helps the reporter to indicate the build-status, though. Anyway good to know. – MakePeaceGreatAgain Dec 11 '20 at 09:32
  • @HimBromBeere is difficult to know how both processes are interacting. Is better to make small stages and keep it simple as test and publish, then run the other process when possible. – sumitani Dec 11 '20 at 15:57