XUnit and MSTest with ExpectedException Returning Different Results

Question

I have a project that is using a yield return and do not understand why XUnit is failing to catch an exception in my unit test while MSTest is passing.

Here is my dummy code.

The bizarre thing is that if I take my private method, EnumerableYieldReturn, and put that logic directly in my public method, YieldReturnList, the outcomes flip with the XUnit test passing and the MSTest failing.

[TestClass]
public class MSTestRunner
{
    [TestMethod]
    [ExpectedException(typeof(ArgumentException))]
    public void MSTestUsingExpectedException()
    {
        var sut = new YieldReturn();

        sut.YieldReturnList(null);
    }
}

public class XUnitRunner
{
    [Fact]
    [ExpectedException(typeof(ArgumentException))]
    public void XUnitUsingExpectedException()
    {
        var sut = new YieldReturn();

        sut.YieldReturnList(null);
    }
}

public class YieldReturn
{
    public IEnumerable<string> YieldReturnList(int? value)
    {
        if (value == null)
            throw new ArgumentException();

        return EnumerableYieldReturn((int)value);
    }

    private IEnumerable<string> EnumerableYieldReturn(int value)
    {
        var returnList = new List<string>() { "1", "2", "3" };

        for (int i = 0; i < value; i++)
        {
            yield return returnList[i];
        }
    }
}

I can get them both to pass by assigning the return object from sut.YieldReturnList and attempting to iterate through it but that doesn't explain why one framework is passing and the other is failing...

numerek · Accepted Answer · 2023-05-11T21:05:25.890

"xUnit.net has done away with the ExpectedException attribute in favor of Assert.Throws." from https://xunit.github.io/docs/comparisons.html.

The reason for the flip of results is that the exception is no longer thrown so:

MSTest: expects the exception because it uses the attribute, and therefore fails because it doesn't get the exception

XUnit: ignores the expect exception attribute because the framework doesn't use it, and therefore passes because an exception didn't cause the test to fail.

The reason the exception is no longer thrown if you change the methods is more complicated but it basically has to do with creating a state machine for methods that use the yield key word. Currently your public method doesn't use the yield key word directly so it is treated like a normal function and therefore executes the null check and throws the exception as soon as the method is called. Moving the yield key word to the public method makes it a lazy state machine so it doesn't execute the null check to throw the exception until you start to iterate the IEnumerable.

Also considering that your public method doesn't return a List you might consider renaming it.

here is how to correctly write the unit tests for the two frameworks

[TestClass]
public class MSTestRunner
{
    [TestMethod]
    [ExpectedException(typeof(ArgumentException))]
    public void MSTestUsingExpectedException()
    {
        var sut = new YieldReturn();

        sut.YieldReturnList(null).Any();
    }
}

public class XUnitRunner
{
    [Fact]
    public void XUnitUsingExpectedException()
    {
        var sut = new YieldReturn();

        Assert.Throws<ArgumentException>(() => sut.YieldReturnList(null).Any());
    }
}

+1. A consequence of the last sentence here is it's useful to break public methods that `yield` into two parts, the first part does validation and the calls the second, which does the `yield`ing. That way the validation happens immediately. — Jon Hanna, Jan 19 '17 at 00:47
+1 for "yield [...] doesn't execute [...] until you start to iterate the IEnumerable". Alternatively, this catches the throw: xUnit: Assert.ThrowAny(() => myEnumerator().Any()) — mtone, May 27 '19 at 06:06
@mtone you are right that I didn't go on to say how you could adjust the code to correctly test for the error being thrown in xUnit. — numerek, Jun 28 '19 at 22:18

XUnit and MSTest with ExpectedException Returning Different Results

1 Answers1