0

I have a Jenkins test agent that is set up to run two distinctly different test jobs that use Nunit3 Console Runner to execute the tests. Each job starts the Nunit3 Console executable on the test agent with a different test assembly. I am trying to run both Jenkins jobs in parallel, so they basically both run Nunit Console to run each of their respective test assemblies.

For some reason, I get a socket error on the second job as shown below. This occurs right after Nunit Console discovers all the tests to run and begins the test execution process:

System.Net.Sockets.SocketException : No connection could be made because the target machine actively refused it 127.0.0.1:12363
10:33:41  
10:33:41  --SocketException
10:33:41  No connection could be made because the target machine actively refused it 127.0.0.1:12363
10:33:41  
10:33:41  Server stack trace: 
10:33:41     at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
10:33:41     at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
10:33:41     at System.Runtime.Remoting.Channels.RemoteConnection.CreateNewSocket(EndPoint ipEndPoint)
10:33:41     at System.Runtime.Remoting.Channels.SocketCache.GetSocket(String machinePortAndSid, Boolean openNew)
10:33:41     at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.SendRequestWithRetry(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream)
10:33:41     at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
10:33:41     at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
10:33:41  
10:33:41  Exception rethrown at [0]: 
10:33:41     at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
10:33:41     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
10:33:41     at NUnit.Engine.ITestAgent.Stop()
10:33:41     at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)
10:33:41     at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
10:33:41     at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
10:33:41     at NUnit.Engine.Runners.MasterTestRunner.Dispose()
10:33:41     at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
10:33:41     at NUnit.ConsoleRunner.Program.Main(String[] args)

Here is the command-line that I am using for the NUnit Console Runner:

nunit3-console.exe testAssemblyDLL --x86 --work=.\TestOutput --timeout=1800000 --trace=Off --result="TestResult.xml" Note that testAssemblyDLL is a different assembly for each process being run.

Is there something missing that I should be doing with the command-line arguments to nunit3-console.exe?

Ken
  • 72
  • 8

1 Answers1

0

The console runner (actually the copy of the test engine associated with it) runs a service to which the launched agent processes subscribe. Running two console copies in parallel may (depending on a lot of factors) cause both of them to use the same endpoint.

An alternative solution would be to use the console runner once, specifying two different assemblies. Each assembly runs by default in it's own process. In the latest versions of the console runner, they will run in parallel by default. In older versions, you might need to specify some options to make that happen. If you indicate the version you are running, I can be more precise.

Charlie
  • 12,928
  • 1
  • 27
  • 31
  • There are two Console Runner versions in-play, one job us using 3.16.3 and the other is using 3.14.0. I can try to make both jobs use the same console runner version if that would help. Because of the way our Jenkins test jobs are set up, it would be difficult to run one instance of the Console Runner with two different assemblies. The testing environments for each job are vastly different, but we want to leverage the same test agent PC to run them. – Ken May 03 '23 at 13:01
  • I'd suggest looking at the command-line passed by each runner to the agent process it creates. If you don't have a TestResult.xml that includes it, you can use --trace:Debug to get the info. If both agents are given the same endpoint to connect to, that's a problem. Note that in development of the runner - at least when I was working on it - allowing more than one copy at a time to run was not an intended feature, so it was not tested. It's likely that you could provide the differing environments you need for each agent process, but it might be a bit of work. :-) – Charlie May 04 '23 at 14:34
  • Both Console runners use the same command-line args (see original post). I guess that I don't understand why the console-runner even needs a TCP endpoint. My tests do not require networking. Or does the runner use IPC using local sockets? If it isn't needed, is there a way that I can tell the console-runner to not use any endpoints? – Ken May 04 '23 at 20:27
  • It's not your command-line I mean but the command-line arguments used by the runner when it launches each agent process. The runners use TCP to communicate with its agents and passes a string representing the endpoint as an argument. – Charlie May 05 '23 at 22:38
  • I see. So, it sounds like this is not within my control at the moment. I am really going to need this to work eventually because we are hoping to utilize our test machines to run multiple console-runners with parallel Jenkins jobs. It would not be feasible to run a single console-runner with multiple test assemblies with the way our testing infrastructure is currently set up. – Ken May 08 '23 at 13:03
  • I suggest you file an issue with the NUnit-Console project. The current team would need to decide whether running executing multiple copies of the runner (as opposed to agent processes) is a design goal they want to incorporate. – Charlie May 09 '23 at 14:25
  • Another workaround idea: run isolated copies in different virtual machines. – Charlie May 09 '23 at 14:26
  • I am testing USB devices on PCs that do not have a lot of resources, so running with Virtual Machines is not feasible either. – Ken May 09 '23 at 18:33