6

I have a series of unit tests that connect to an Azure Storage emulator. In Setup my code checks if there is something listening on the emulator's port, and if not sets a flag StorageNotAvailable.

In each of my tests I have some code...

if ( StorageNotAvailable )
    Assert.Inconclusive( "Storage emulator is not available" )

// where storage emulator is available, continue as normal

As expected, when the test returns void this reports correctly in the Test Explorer as "Inconclusive".

When the test is exercising some async methods, and the [TestMethod] signature returns Task then the test is reported in the TestExplorer as "Failed" instead of "Inconclusive".

How can I get an async method to report as Inconclusive?

EDIT

Some additional detail may be in order. Here are some sample tests I rigged up to demonstrate the problem I am seeing.

[TestMethod]
public void MyTestMethod()
{
    Assert.Inconclusive( "I am inconclusive" );
}

[TestMethod]
public async Task MyTestMethodAsync()
{
    Assert.Inconclusive( "I am an error" );
}

Image of the offending code in action

Test Explorer Error

Test Explorer Inconclusive

Some environment details may be in order as well:

  • Windows 10 x64 1703 Build 15063.608
  • Visual Studio Enterprise 2017 15.3.5
  • .NET 4.7.02046
  • VS Extensions that might be relevant
    • Microsoft Visual Studio Test Platform
    • MSTest V2 Create Unit Test Extension
    • MSTest V2 IntelliTest Extension
    • MSTest V2 Templates
  • Project references that might be relevant
    • Microsoft.VisualStudio.TestPlatform.TestFramework v14.0.0.0
    • Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions v14.0.0.0
  • Project nuGet packages that might be relevant
    • MSTest.TestAdapter v1.1.18
    • MSTest.TestFramework v1.1.18
  • Project build target is .NET Framework v4.7
jessehouwing
  • 106,458
  • 22
  • 256
  • 341
Richard Hauer
  • 1,298
  • 9
  • 22
  • I took the exact code from your examples and again as expected it passed. So I am at a loss as to why you are have that experience with the assertion. Literally copy, paste, run. Only difference being that I am not using VS enterprise but community – Nkosi Sep 26 '17 at 14:32
  • Same versions & all? Could this be a VS bug? – Richard Hauer Sep 26 '17 at 14:39
  • community 15.3.26730.16 – Nkosi Sep 26 '17 at 14:40
  • went back to community 2015 and also 2013 and works. all using mstest – Nkosi Sep 26 '17 at 14:41
  • Well i could suggest repairing the current VS installation and trying the test(s) again. – Nkosi Sep 26 '17 at 14:42
  • I can reproduce it on 2 machines (both Enterprise). Empty Soln. New Test Project. Paste code. Bang. – Richard Hauer Sep 26 '17 at 14:47
  • Download community on another fresh box (if available) and see if you get the same problem. could be version specific. Do you have the latest updates installed (if any) – Nkosi Sep 26 '17 at 14:48
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/155343/discussion-between-richard-hauer-and-nkosi). – Richard Hauer Sep 26 '17 at 14:53
  • Verified in Community Edition. Verified with all VS "test" Extensions disabled. Verified with v1.1.17 and v1.1.18 of MSTest.TestFramework – Richard Hauer Sep 26 '17 at 15:09

2 Answers2

5

Assert.Inconclusive raises a special kind of exception, which will cause the task to catch that exception. Since the Task library and async don't know about, we can't blame them for complaining. The Task framework will wrap the exception in an AggregateException, which I suspect is getting reported. This was a nice assumption, but it turned out that the code looking for AssetInconclusiveException was comparing the raised instance against MstestV1's implementation and not MsTestV2.

But I suppose this should be considered a bug in the MsTest v2 runner, which should inspect all Tasks that failed and look at the exception that caused their failure.

The behaviour is a known behaviour at the moment and I've just submitted a PR to fix this. Pull Request Merged, now just wait for the next Nuget build to trigger.

This fix was merged and released in the latest 1.2.0 package.

jessehouwing
  • 106,458
  • 22
  • 256
  • 341
  • Thanks for confirming this - I went looking for this bug but my Google-fu (or Bing-fu?) wasn't up to the job apparently. Glad I'm not going mad. – Richard Hauer Sep 27 '17 at 23:40
0

This is caused by a confirmed bug in MsTest.Framework code -- issue #249 on GitHub is tracking the problem and eventual solution:

Async tests which use Assert.Inconclusive report as Error

https://github.com/Microsoft/testfx/issues/249

Jorgen Thelin
  • 1,066
  • 9
  • 23