I seem to have an issue when running a PowerShell script as a part of a step in our deployment Octopus pipeline. The tests run against a few websites, and each website has a range of tests. These are just locally setup websites on our environment.
The website with the most tests runs has about 100 tests, and the smallest website is about 5. The issue can occur with the website with 5 or 100. These are SpecFlow tests.
The results are then exported to an xml file. According to the xml file it does look like it has ran all the tests, as it shows the ones that succeeded and failed. It doesn't seem like this issue always occurs. Although it happens if I leave it on the nightly build. It won't always occur if I trigger a manual build on Octopus.
The version of NUnit is 3.1.1
Here is so far an example line of what I am passing in powershell for the NUnit console.
param(
[string]$configurationDescription = $OctopusParameters['environment.Transform'] + " Env Test Run",
[string]$site,
[string]$environmentName,
[string]$environmentSiteTag,
[string]$testDllName,
[string]$testDllFolder
)
$folderPath = $OctopusParameters['tests.rootFolder']
Set-Location $folderPath
$configurationDescription = $configurationDescription.Replace(" ", "-")
$testResultsFolderName = "testResults"
$testResultsPath = "$folderPath\$testResultsFolderName\$site"
$currentTimeFormatted = Get-Date -Format "dd-MM-yyyy_hh_mm_ss"
if(!(Test-Path -Path $testResultsPath))
{
New-Item -Path $testResultsPath -ItemType Directory
}
$testDllPath = $folderPath + "\tests\$testDllFolder\$testDllName"
$environmentWipTag = $OctopusParameters["tests.environment.wipname"]
#Change the location of "bin\XXX" So it will pick up the respective dll file and use the respective App.XXX.config
$testResultsXmlPath = "$testResultsPath\TestResult_$currentTimeFormatted.xml"
$args = "$testDllPath --where `"cat != $environmentWipTag && (cat == $environmentName && cat == $environmentSiteTag)`" --result=$testResultsXmlPath --framework=net-4.0 --timeout=20000"
# SECTION: Update Chrome driver
# Update Chrome Driver for testing to match the machine.
#1. Get the version of Chrome driver installed.
$chromeDriverOutputPath = $folderPath + "\tests\$testDllFolder"
# Store original preference to revert back later
$OriginalProgressPreference = $ProgressPreference;
# Increases the performance of downloading the ChromeDriver.
$ProgressPreference = 'SilentlyContinue';
try
{
$chromeVersion = (Get-Item (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\chrome.exe').'(Default)').VersionInfo.FileVersion
$chromeVersion = $chromeVersion.Substring(0, $chromeVersion.LastIndexOf("."));
} catch
{
"Could not find Google Chrome in the registry."
}
#2. Get the latest version of chrome driver available.
$chromeDriverVersion = (Invoke-WebRequest "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_$chromeVersion" -UseBasicParsing).Content;
Write-Host "Latest matching version of Chrome Driver is $chromeDriverVersion";
#3.Create a temp folder to store the chrome driver to be downloaded.
$tempFilePath = [System.IO.Path]::GetTempFileName();
$tempZipFilePath = $tempFilePath.Replace(".tmp", ".zip");
Rename-Item -Path $tempFilePath -NewName $tempZipFilePath;
$tempFileUnzipPath = $tempFilePath.Replace(".tmp", "");
# 4. Download correct chrome drive, unzip the archive and move the chromedriver to the correct location.
Invoke-WebRequest "https://chromedriver.storage.googleapis.com/$chromeDriverVersion/chromedriver_win32.zip" -OutFile $tempZipFilePath -UseBasicParsing;
Expand-Archive $tempZipFilePath -DestinationPath $tempFileUnzipPath;
Move-Item "$tempFileUnzipPath/chromedriver.exe" -Destination $chromeDriverOutputPath -Force;
# 5. Clean up files.
Remove-Item $tempZipFilePath;
Remove-Item $tempFileUnzipPath -Recurse;
# Set back to default ProgressPreference.
$ProgressPreference = $OriginalProgressPreference
#END SECTION: Update Chrome driver
Write-Host "Chrome Driver now matches the version installed on the machine. Beginning Test run."
#Begin Test Run.
$nunitRunnerResult = (Start-Process -FilePath "$folderPath\runner\nunit3-console.exe" -ArgumentList $args -PassThru -Wait).ExitCode
if(!(Test-Path -Path $testResultsXmlPath))
{
Write-Host "Args:$args FilePath:$folderPath\runner\nunit3-console.exe"
Write-Error "Unable to find $testResultsXmlPath"
return;
}
$testsTestResultHtmlPagePath = "$testResultsPath\$configurationDescription-$currentTimeFormatted\ExtentReport"
$args = "-i $testResultsXmlPath -o $testsTestResultHtmlPagePath -r html"
Start-Process -FilePath "$folderPath\extentreport\extent.exe" -ArgumentList $args -Wait
$extentReportPath = $testsTestResultHtmlPagePath + "\index.html"
$extentSummaryPath = $testsTestResultHtmlPagePath + "\dashboard.html"
$testsTestResultPath = "$testResultsFolderName\testsTestResult-$currentTimeFormatted.xml"
$args = [string]::Format("-f=Features/ -o=$testResultsFolderName/{0}-{1}/PicklesReport -trfmt=nunit3 -df=dhtml -lr={2}", $configurationDescription, $currentTimeFormatted, $testsTestResultPath)
Exit 0
The C# ran after each test.
using BoDi;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using System;
using System.Configuration;
using TechTalk.SpecFlow;
namespace MyTestFramework.Helpers
{
[Binding]
public sealed class Hooks
{
private readonly IObjectContainer _container;
private IWebDriver _webDriver = default;
public Hooks(IObjectContainer objectContainer)
{
_container = objectContainer;
}
[BeforeScenario]
public void BeforeTestRun()
{
InstantiateDriver();
}
[AfterScenario]
public void AfterScenario()
{
if (!(_webDriver is null))
{
//Close any additional tabs open and then delete all cookies when one tab is open
var tabs = _webDriver.WindowHandles;
int numberOfTabs = _webDriver.WindowHandles.Count;
while (numberOfTabs > 1)
{
_webDriver.SwitchTo().Window(tabs[numberOfTabs - 1]);
_webDriver.Close();
numberOfTabs = _webDriver.WindowHandles.Count;
_webDriver.SwitchTo().Window(tabs[numberOfTabs - 1]);
}
_webDriver.Manage().Cookies.DeleteAllCookies();
_webDriver.Close();
_webDriver.Quit();
_webDriver.Dispose();
}
}
[AfterTestRun]
public static void After()
{
}
private void InstantiateDriver()
{
var selectedDriver = ConfigurationManager.AppSettings.Get("selectedWebDriver");
if (string.IsNullOrEmpty(selectedDriver))
{
throw new ArgumentException(Constants.ExceptionMessages.SelectedWebdriverCouldNotBeFound, selectedDriver);
}
switch (selectedDriver)
{
case "Chrome":
_webDriver = new ChromeDriver();
break;
case "Chrome Headless":
var chromeOption = new ChromeOptions();
chromeOption.AddArguments("headless");
chromeOption.AddArguments("--window-size=1680,1050");
_webDriver = new ChromeDriver(chromeOption);
break;
default:
throw new ArgumentException("Webdriver could not be found", selectedDriver);
}
_webDriver.Manage().Window.Maximize();
_container.RegisterInstanceAs<IWebDriver>(_webDriver);
}
}
}
If I close the chrome driver it must close everything as the Octopus step then seems to fail. Worth noting as well this does not happen if directly using visual studio.
