1

We have an ASP.NET Core 3.0 application hosted behind IIS on Windows Server 2016. Under heavy load (5,000+ concurrent users) we start to get 503 errors with the message "The service is unavailable."

I checked the logs at %SystemRoot%\System32\LogFiles\HTTPERR and realized they are mostly due to 'LimitExceeded'. Here's some sample data:

2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 205 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 207 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 209 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 211 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 213 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 215 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 217 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 219 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 221 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 223 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 225 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 227 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 229 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 231 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 233 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 235 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 237 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 239 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 241 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 243 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 245 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 247 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 249 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 251 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 253 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 255 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 257 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 259 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 261 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 263 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 265 - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - 267 - - LimitExceeded -
2019-11-22 20:36:33 5.113.159.51 47537 185.211.56.203 443 - - - - - - Client_Reset -
2019-11-22 20:36:33 5.125.149.3 54861 185.211.56.203 443 HTTP/2.0 GET /api/messaging/checkcallcompatibility 72959 - 7 Client_Reset patogh.me(domain)(4.0)(pool)
2019-11-22 20:36:33 5.112.111.38 36112 185.211.56.203 443 - - - - - - ClientCancel -
2019-11-22 20:36:33 5.121.34.242 55374 185.211.56.203 443 HTTP/2.0 GET /api/messaging/checkcallcompatibility 94051 - 7 Client_Reset patogh.me(domain)(4.0)(pool)
2019-11-22 20:36:33 83.122.43.131 34801 185.211.56.203 443 - - - - - - LimitExceeded -
2019-11-22 20:36:33 5.119.159.122 51429 185.211.56.203 443 - - - - - - LimitExceeded -
2019-11-22 20:36:33 5.74.28.250 12622 185.211.56.203 443 - - - - - - Client_Reset -
2019-11-22 20:36:33 5.112.111.38 36113 185.211.56.203 443 - - - - - - Client_Reset -
2019-11-22 20:36:33 89.196.176.185 38334 185.211.56.203 443 HTTP/2.0 GET /api/messaging/checkcallcompatibility 4921 - 7 Client_Reset patogh.me(domain)(4.0)(pool)
2019-11-22 20:36:34 5.121.34.242 55374 185.211.56.203 443 HTTP/2.0 GET /api/messaging/checkcallcompatibility 94065 - 7 Client_Reset patogh.me(domain)(4.0)(pool)
2019-11-22 20:36:34 91.251.90.58 43009 185.211.56.203 443 HTTP/2.0 GET /api/messaging/checkcallcompatibility 68999 - 7 Client_Reset patogh.me(domain)(4.0)(pool)
2019-11-22 20:36:34 5.112.111.38 36115 185.211.56.203 443 - - - - - - Client_Reset -
2019-11-22 20:36:34 172.80.200.76 56406 185.211.56.203 443 - - - - - - Client_Reset -

So how do I know which limit is exceeded and how can I change that limit?

SepehrM
  • 111
  • 4

1 Answers1

0

After wasting a week and loosing lots of money, stumbled upon this tutorial which dates back to 2012.

https://github.com/SignalR/SignalR/wiki/Performance

Apparently there's a ridiculous 5,000 limit on max concurrent requests per application that IIS handles. So I ran the command below at %windir%\System32\inetsrv\ changing it to 50,000.

appcmd.exe set config /section:system.webserver/serverRuntime /appConcurrentRequestLimit:50000

This limit might have been high enough when web sockets were not a thing and servers were not powerful enough to handle more requests. Nowadays most servers can handle tens of thousands of concurrent requests quite easily and many applications require persistent web socket connections.

SepehrM
  • 111
  • 4