2

We have recently implemented the backplane using sql server. The server that we created the backplane contains other databases as well. These other databases are accessed by different applications which are on different servers. For stress testing we implemented a simple signalr client program which sends a message to the server every 30 seconds. There are two servers which are handled by a load balancer. The backplane works perfectly when number of clients are small.

The issue is when the number of clients are higher. (closer to 50 or more). The following exception is thrown by other applications that tries to use the database server for a different database.

System.Data.SqlClient.SqlException: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
   --- End of inner exception stack trace ---
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
   at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__5()
   at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at System.Linq.Queryable.Count[TSource](IQueryable`1 source)

Because of this exception all the other applications breaks since they can not access the database. The problem immidiately fixes if we disable the backplane. (stopping the two signalr servers). Is this the expected behavior when using the backplane with high number of users or is it an issue with the backplane?

PS - We noticed that the machines that throw these exceptions have more than 3000 threads allocated for w3wp.exe process. Seems like they are stuck on a operation.

enter image description here

enter image description here

Janitha Tennakoon
  • 856
  • 11
  • 40
  • Have you tried to profile SQL server and find out is there any heavy queries that take more than 30 sec to finish? Any performance investigations? – cassandrad May 03 '17 at 12:26
  • Unfortunately we do not have access for that database machine, but the same queries that gets timeout exception works perfectly fine when we disable the backplane. – Janitha Tennakoon May 04 '17 at 03:15
  • Normally, backplane should not cause any troubles. Moreover, SQL server is able to survive more huge workload. Have you checked amount of connections and sending messages? – cassandrad May 04 '17 at 08:07
  • We found out that w3wp.exe in the other application servers (not in the signalr server) have more than 3000 threads allocated. Seems like they are stuck on a operation. – Janitha Tennakoon May 04 '17 at 09:35
  • Definitely, you should get possibility to profile servers' activity. Otherwise it is almost impossible to provide any help. – cassandrad May 04 '17 at 09:40

1 Answers1

1

If you are not using asynchronous calls then its feasible that one of the message threads are taking longer than expected, and while it is waiting to return another thread is stuck and you are basically getting threadlocked.

The 2 possible solutions I can think of is:

1) Make sure you are making async calls to the database.

2) Make sure you have enough threads setup on the IIS server. You need a minimum of 12, default setup is usually only 1.

Kelso Sharp
  • 972
  • 8
  • 12