AppVeyor + OpenCover + xUnit - Build doesn't fail when tests fail

Question

I created an AppVeyor Build Script that uses OpenCover and Coveralls.Net to run my xUnit tests and publish the code coverage to Coveralls.io.

But when I have failing tests - AppVeyor reports the build successful. How do I configure AppVeyor to fail if OpenCover + xUnit report failing tests?

The script is based on csMACnz's sample:

.\src\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe 
   -register:user 
   -target:"xunit.console.clr4.exe"  
   "-targetargs:""src\HttpWebRequestWrapper.Tests\bin\$env:CONFIGURATION\HttpWebRequestWrapper.Tests.dll"" 
   /noshadow 
   /appveyor" 
   -filter:"+[HttpWebRequestWrapper*]*" 
   -output:opencoverCoverage.xml

    $coveralls = (Resolve-Path "src/packages/coveralls.net.*/tools/csmacnz.coveralls.exe").ToString()
    & $coveralls --opencover -i opencoverCoverage.xml --repoToken $env:COVERALLS_REPO_TOKEN --commitId $env:APPVEYOR_REPO_COMMIT --commitBranch $env:APPVEYOR_REPO_BRANCH --commitAuthor $env:APPVEYOR_REPO_COMMIT_AUTHOR --commitEmail $env:APPVEYOR_REPO_COMMIT_AUTHOR_EMAIL --commitMessage $env:APPVEYOR_REPO_COMMIT_MESSAGE --jobId $env:APPVEYOR_JOB_ID

I've tried adding the -returntargetcode flag to the OpenCover.Console.exe code, but that doesn't seem to signal AppVeyor to fail the build.

score 1 · Accepted Answer · answered Feb 08 '18 at 23:11

I think the reason is simple. Following how you wrote the YML file, AppVeyor expects the whole test_script to provide a non-zero return code, while a failure inside the PowerShell script section you left won't lead to such a case.

You have to run each commands separately, and then any failure will fail the whole build, as my AppVeyor script shows

test_script:
  - dotnet test Tests\CSharpCore\Tests.NetStandard.csproj --configuration Release --no-build --filter "FullyQualifiedName~Lextm.SharpSnmpLib.Integration.TrapDaemonTestFixture"
  - dotnet test Tests\CSharpCore\Tests.NetStandard.csproj --configuration Release --no-build --filter "FullyQualifiedName~Lextm.SharpSnmpLib.Integration.DaemonTestFixture"
  - dotnet test Tests\CSharpCore\Tests.NetStandard.csproj --configuration Release --no-build --filter "FullyQualifiedName~Lextm.SharpSnmpLib.Unit"

If I wrap the three commands in a batch file or PowerShell script, I can reproduce the same issue you experienced.

interesting - does this mean if the first test run fails (`TrapDaemonTestFixture`) will the other tests not run? (ie `DaemonTestFixture` and `Unit`)? — Philip Pittle, Feb 08 '18 at 23:21
@PhilipPittle my case is special that the test cases are resource consuming, so I have to split them to small batches. But yes, if a command fails, then the next few commands won't be executed at all, as AppVeyor will break from there. — Lex Li, Feb 09 '18 at 01:50

score 1 · Answer 2 · answered Feb 09 '18 at 00:53

Agree with Lee, but simpler option would be having $ErrorActionPreference = "Stop" in the beginning of your script.

Looking it your comment to the post below, you need all commands to be executed even if something failed. I this case look at ErrorVariable. Good discussion and sample is here.

score 0 · Answer 3 · answered Feb 09 '18 at 02:22

Looks like there are a few ways to get the desired behavior out of AppVeyor. I ended up using $LastExitCode referenced in the AppVeyor Discussion Thread @ilyaf posted.

This approach ends up giving me a bit more flexibility - I could continue to run the entirety of my testing script but still fail the testing step if any specific piece failed:

.\src\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe 
  -register:user 
  -target:"xunit.console.clr4.exe"  
  "-targetargs:""src\HttpWebRequestWrapper.Tests\bin\$env:CONFIGURATION\HttpWebRequestWrapper.Tests.dll"" 
  /noshadow 
  /appveyor" 
  -filter:"+[HttpWebRequestWrapper*]*" 
  -output:opencoverCoverage.xml

$testRunnerErrorCode = $LastExitCode

#can move this to end of script if I want to still publish code coverage
#for failed tests
if ($testRunnerErrorCode -ne 0){
   throw "xUnit failed with code $testRunnerErrorCode "
}


$coveralls = (Resolve-Path "src/packages/coveralls.net.*/tools/csmacnz.coveralls.exe").ToString()
& $coveralls --opencover -i opencoverCoverage.xml --repoToken  $env:COVERALLS_REPO_TOKEN --commitId $env:APPVEYOR_REPO_COMMIT --commitBranch $env:APPVEYOR_REPO_BRANCH --commitAuthor $env:APPVEYOR_REPO_COMMIT_AUTHOR --commitEmail $env:APPVEYOR_REPO_COMMIT_AUTHOR_EMAIL --commitMessage $env:APPVEYOR_REPO_COMMIT_MESSAGE --jobId $env:APPVEYOR_JOB_ID

I'm going to mark Lex's answer correct as it's the most straight forward and direct answer to the question.

AppVeyor + OpenCover + xUnit - Build doesn't fail when tests fail

3 Answers3