Can I Use NUnit & NCrunch to Test Multiple Build Configurations with Preprocessor Directives?

Question

I am writing a wrapper library for the API to some software that has multiple versions of the API, and lots of shared calls between multiple platforms that are developed concurrently but separately. Over time they have even been merging the platforms to use the same code base, just under different namespaces & *.exe builds.

I'm doing this by writing one single code base, and then using preprocessor directives and conditional compilation symbols through build configurations to selectively use certain code for the builds. About 90% of the code can literally be reused between versions and platforms, so this is helpful. All works fine on the project-under-test end.

However, I am having issues using NUnit & NCrunch to unit test this project. I created identical build configurations to load the correct constants and create the correct build folders for the Integration Test project. However, I am noticing two strange problems:

1. NUnit seems to be ignoring preprocessor directives in the Integration Test project. For example, in the following example, the first line of code is always hit (BUILD_Foov16=true) regardless of configuration (e.g. BUILD_Bar_2015=true), even though in Visual Studio it appears that the desired line set (corresponding to the current configuration variables) is the only one active:

   [TestFixture]
   public class FooBarIncApplicationTests
   {
      #if BUILD_Foov16
           public const string path = @"C:\Program Files (x86)\FooBarInc\FooV16\Foo.exe";
      #elif BUILD_Foov17
           public const string path = @"C:\Program Files (x86)\FooBarInc\FooV17\Foo.exe";
      #elif BUILD_Bar_2013
           public const string path = @"C:\Program Files (x86)\FooBarInc\Bar 2013\Bar.exe";
      #elif BUILD_Bar_2015
           public const string path = @"C:\Program Files (x86)\FooBarInc\Bar 2015\Bar.exe";
      #endif
   
      [Test]
      public void FooBarIncApplication_Initialize_New_Instance_Defaults()
      {
         using (FooBarIncApplication app = new FooBarIncApplication(path))
         {
         ...
         }
      }
   }
   

2. Additionally, it seems that when I run a test through NCrunch, it is only using the *.dll corresponding to the build created by the first configuration listed (e.g. it is always testing the *.dll compiled for Foo.exe v16.

It seems to me like these two issues are related. I am wondering if NUnit and/or NCrunch cannot handle such a setup, or if there is a particular way I should be dealing with this unique setup?

My bigger issue is #2, which is that NCrunch seems to be running NUnit only on the *.dll built from the first configuration, which makes it impossible to test any of the other configurations. Perhaps this is a problem with project dependencies? (The path in the example above is to the program that I am interacting with via the API, but not my *.dll projects.)

Answer 1

OK, I found out what the problem was. This is worth knowing if you are ever testing NCrunch for multiple DLL build configurations!

When NCrunch runs, the *.dll generated and the code coverage ascertained (and stepped through) is NOT affected by the current configuration specified in Visual Studio. It is always determined by the project default configuration at the time that Visual Studio loads the project. This means that in order to change configurations for testing, you need to modify the *.csproj file.

For example, I have configuration definitions for Foov16 and Foov17 shown below. In order to set the project to Foov17 in such a way that NCrunch will work, the configuration for Foov17 must be referenced in the default configuration (the first element):

<PropertyGroup>
   <Configuration Condition=" '$(Configuration)' == '' ">Debug-Foov17</Configuration>
   <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
   <ProjectGuid>{...}</ProjectGuid>
   <OutputType>Library</OutputType>
   <AppDesignerFolder>Properties</AppDesignerFolder>
   <RootNamespace>FooBarInc.API</RootNamespace>
   <AssemblyName>FooBarInc.API</AssemblyName>
   <TargetFrameworkVersion>v4.5.2</TargetFrameworkVersion>
   <FileAlignment>512</FileAlignment>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug-Foov16|AnyCPU'">
   <DebugSymbols>true</DebugSymbols>
   <OutputPath>bin\AnyCPU\Debug-Foov16\</OutputPath>
   <DefineConstants>TRACE;DEBUG;BUILD_Foov16</DefineConstants>
   <DebugType>full</DebugType>
   <PlatformTarget>AnyCPU</PlatformTarget>
   <ErrorReport>prompt</ErrorReport>
   <CodeAnalysisRuleSet>MinimumRecommendedRules.ruleset</CodeAnalysisRuleSet>
   <DocumentationFile>bin\AnyCPU\Debug-Foov16\FooBarInc.API.XML</DocumentationFile>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug-Foov17|AnyCPU'">
   <DebugSymbols>true</DebugSymbols>
   <OutputPath>bin\AnyCPU\Debug-Foov17\</OutputPath>
   <DefineConstants>TRACE;DEBUG;BUILD_Foov17</DefineConstants>
   <DebugType>full</DebugType>
   <PlatformTarget>AnyCPU</PlatformTarget>
   <ErrorReport>prompt</ErrorReport
   <CodeAnalysisRuleSet>MinimumRecommendedRules.ruleset</CodeAnalysisRuleSet>
   <DocumentationFile>bin\AnyCPU\Debug-Foov17\FooBarInc.API.XML</DocumentationFile>
</PropertyGroup>

Answer 2

Not sure if that helps you. If I understood you correctly you want to inject different exe paths in a test case and let the test case run for every exe path. You can try to use NUnits TestCaseattribute for this. Example from NUnit website:

   [TestCase(12,3,4)]
   [TestCase(12,2,6)]
   [TestCase(12,4,3)]
   public void DivideTest(int n, int d, int q)
   {
     Assert.AreEqual( q, n / d );
   }

Based on that you may rewrite your test to:

[TestCase(@"C:\Program Files (x86)\FooBarInc\FooV16\Foo.exe")]
[TestCase(@"C:\Program Files (x86)\FooBarInc\FooV17\Foo.exe")]
public void FooBarIncApplication_Initialize_New_Instance_Defaults(string path)
{
    using (FooBarIncApplicatio app = new FooBarIncApplicatio(path))
    {
    ...
    }
 }

Base on that the testcase will run with version 16 and version 17 of your exe. Additional benefit: You don't have to switch your VS configurations to run all tests.

If you still want to use conditional test case execution you can either use the Category or the System.Diagnostics.Conditional attribute. See this stackoverflow link for additional information.

Hope that helps.