3

I am trying to track down a bug in my Xunit acceptance tests on Azure Pipelines and want to enable stoponfail to make the problem easier to track down. It's an intermittent timing thing, so hard to reproduce running tests locally.

I have the following in my yml for Azure Pipelines 

      - task: DotNetCoreCLI@2
        displayName: dotnet test
        inputs:
          command: test
          arguments: '--configuration $(BuildConfiguration) --no-build --verbosity $(Verbosity)'

It sounds like the only way to make Xunit stop on first failure is with the -stoponfail argument. Unfortunately, this isn't compatible with dotnet test. Further, if I switch to running dotnet xunit instead of dotnet test I can no longer use my existing build script and need to install the dotnet xunit extension during my build pipeline. Even further - it doesn't look like dotnet xunit will emit trx files, which makes it fairly useless in the context of a CI pipeline.

What are other people doing to be able to use Xunit and stop on failure?

Jeff
  • 35,755
  • 15
  • 108
  • 220
  • Why can't you use `VSTest@2 `? (as long as you have the XUnit Test Adapter in any project it will use XUnit). – Erik Philips Jul 09 '20 at 01:25
  • How would that allow me to pass stoponfail? – Jeff Jul 10 '20 at 02:21
  • A test that fails stops the pipeline by default. In order for a failed test task to continue, one would have to enable `Continue on error`. If you mean don't continue to run tests after the first failure, it's not an option. It's not really a scale-able option so it's not really supported (how does that work on parallel testing?). – Erik Philips Jul 10 '20 at 02:50
  • It you have a chain of acceptance tests with an order specified (in which case parallelism isn’t an issue), it makes it extremely hard to debug a failed test if each test after it just continues on running. In situations other than acceptance tests (unit tests, integration tests), I would agree with you. If there is another pattern for organizing long running acceptance tests other than by ordering individual test methods and classes, I’m open to that too. However my schooling on test design is generally to limit to one assertion per test method. – Jeff Jul 10 '20 at 03:56

0 Answers0