Specflow Test with Data Tables runs from Agent or Client Fails on Azure Pipeline VS Test Task

Question

BLUF: Without Specflow Data Table tests run fine from Azure Pipeline With Specflow Data Table tests fail from Azure Pipeline

Setup

1. SpecFlow 3.1 or 3.3
2. MSTest as Test Runner
3. packages.config
4. .feature.cs files are generated using SpecFlow.Tools.MsBuild.Generation NuGet package
5. VS 2019
6. SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings Disabled
7. .NET Framework 4.5.1 (Framework on Self-hosted Machine .NET 4.7)
8. TFS/VSTS/Azure DevOps – Task – Visual Studio Test Task 2.*

Configuration

1. Visual Studio C# Selenium Test Project
2. Uses Extent Reports
3. DotNetSeleniumExtras Page Objects
4. MSTestTestAdapter and Framework
5. Selenium and Chrome Driver
6. Data Tables in the Feature File
7. Azure Visual Studio Tasks on Build or Release Pipeline
8. Test Assembly Runs with an .exe and not a .dll
9. Azure VSTest Task uses testhost.x86.exe

Actions:

1. Create a Selenium Specflow Test with a Data Table in Visual Studio
2. Push to a Repo
3. Set up a Build or Release Pipeline with a Visual Studio Test Task (Test acts the same way on either)
4. Run the Pipeline

Results:

1. Tests with or without data tables run without error from the client machine via Visual Studio Test Explorer
2. Tests with or without data tables run without error from the custom local agent machine via Visual Studio Test Explorer or when run with vstest.console.exe from the command line
3. Tests without data tables run without error from the custom local agent machine via Visual Studio Test Explorer or when run with vstest.console.exe from the command line

Tests with data tables fail with the following error from an Azure Pipeline Visual Studio Task

dmp file error: Process Name: testhost.x86.exe : C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\Extensions\TestPlatform\testhost.x86.exe Process Architecture: x86 Exception Code: 0xC0000005 Exception Information: The thread tried to read from or write to a virtual address for which it does not have the appropriate access. Heap Information: Not Present

System Information

• OS Version: 10.0.14393
• CLR Version(s): 4.7.3620.0

Azure PipelineError output:

Test method HappyTest.Features.CPUF.YPUCC_1 threw exception: 
System.NullReferenceException: Object reference not set to an instance of an object.

AventStack.ExtentReports.Model.ExceptionInfo..ctor(Exception ex)

AventStack.ExtentReports.ExtentTest.Log(Status status, Exception ex, MediaEntityModelProvider provider)

AventStack.ExtentReports.ExtentTest.Fail(Exception ex, MediaEntityModelProvider provider)

UtilityLibrary.Utilities.Hooks.Hooks.InsertReportingSteps(ScenarioContext InjectedText) in D:\DevOpsAgent\_work\8\s\UtilityLibrary\Utilities\Hooks\Hooks.cs: line 188 lambda_method(Closure , IContextManager , ScenarioContext )

TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration) in D:\a\1\s\TechTalk.SpecFlow\Bindings\BindingInvoker.cs: line 69

TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.InvokeHook(IBindingInvoker invoker, IHookBinding hookBinding, HookType hookType) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs: line 351

TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.FireEvents(HookType hookType) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs: line 340

TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.FireScenarioEvents(HookType bindingEvent) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs: line 321

TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnStepEnd() in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs: line 304

TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs: line 470

TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.Step(StepDefinitionKeyword stepDefinitionKeyword, String keyword, String text, 
String multilineTextArg, Table tableArg) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs: line 581

TechTalk.SpecFlow.TestRunner.Given(String text, String multilineTextArg, Table tableArg, String keyword) in D:\a\1\s\TechTalk.SpecFlow\TestRunner.cs: line 79

HappyTest.Features.CPUF.YPUCC(Anonymized Data Fields Here) in D:\DevOpsAgent\_work\8\s\HappyTest\Features\CreatePersonalUser.feature: line 12

HappyTest.Features.CPUF.YPUCC_1() in D:\DevOpsAgent\_work\8\s\HappyTest\Features\CreatePersonalUser.feature: line 11

-> Loading plugin D:\DevOpsAgent\_work\8\s\HappyTest\bin\Release\TechTalk.SpecFlow.MSTest.SpecFlowPlugin.dll

-> Using app.config

Given The user navigates to Sbn Test
:: The page type is HappyTest.Pages.BasePage
:: Sbn Test Page Displayed
-> done: BaseScenarioSteps.GivenUserNavigatesToSbnTest() (2.0s)
Given The user selects Personal Subscribe Yearly
:: The page type is HappyTest.Pages.SbnHome
-> error: Timed out after 10 seconds
:: WebDriver Stopped

YAML

    pool:
      name: qa-automation
      demands:
      - msbuild
      - visualstudio
    
    variables:
      BuildPlatform: ''
      BuildConfiguration: 'Debug'
    
    steps:
    - task: NuGetToolInstaller@1
      displayName: 'Use NuGet 5.3.1'
      inputs:
        versionSpec: 5.3.1
    
    - task: NuGetCommand@2
      displayName: 'NuGet restore'
      inputs:
        restoreSolution: SeleniumTest.sln
    
    - task: VSBuild@1
      displayName: 'Build solution SeleniumTest.sln'
      inputs:
        solution: SeleniumTest.sln
        platform: '$(BuildPlatform)'
        configuration: '$(BuildConfiguration)'
    
    - task: PublishSymbols@2
      displayName: 'Publish symbols path'
      inputs:
        PublishSymbols: false
    
    - task: CopyFiles@2
      displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
      inputs:
        SourceFolder: '$(system.defaultworkingdirectory)'
        Contents: '**\bin\$(BuildConfiguration)\**'
        TargetFolder: '$(build.artifactstagingdirectory)'
        CleanTargetFolder: true
        OverWrite: true
    
    - task: PublishBuildArtifacts@1
      displayName: 'Publish Artifact: drop'
    
    - task: VisualStudioTestPlatformInstaller@1
      displayName: 'Visual Studio Test Platform Installer'
      inputs:
        versionSelector: latestStable
    
    - task: VSTest@2
      displayName: 'VsTest - testAssemblies'
      inputs:
        testAssemblyVer2: |
         !**\*TestAdapter.dll
         **\HappyTest.exe
         !**\obj\**
        runTestsInIsolation: true
        otherConsoleOptions: '/Tests:Happy_1'
        testRunTitle: 'Validate Specflow Data Table Tests'
        diagnosticsEnabled: true
        collectDumpOn: always

VS Actions Tried Unsuccessfully

1. Updated SpecFlow to Beta 3.3
2. Updated Selenium and most other packages
3. Updated Dependencies in app.config
4. Updated .NET Framework to 4.6.2 then reverted
5. Added Additional waits before and after browser rendering
6. Tried toggling between x86 vs x64 vs any cpu architecture on Build Platform
7. Used full app.config
8. Added 60 second WebDriverWait

Azure Actions Tried unsuccessfully:

1. Changed multiple parameters on the Build and Release Tasks
2. Changed all variables from release to debug
3. Tried tests on both Azure and Local Agents
4. Changed Visual Studio versions from 2017 to 2019 and latest on the VS Build Task
5. Checked Run Tests in isolation
6. Pushed Packages Folder to Release Drop