19

What is the Http queue length metric on Azure Web App ?

My Web App is constantly over 150.

I'm concerned since the default Alert that can be enabled on the App Service has a default threshold of 100.

Could the usage of SignalR affect this metric ?

EDIT:

Here is typical day load: enter image description here

Guillaume Morin
  • 3,910
  • 6
  • 25
  • 40
  • 1
    This is addressed in a ServerFault answer - question [here](http://serverfault.com/questions/730721/windows-azure-autoscaling-other-than-cpu-usage). – David Makogon Nov 22 '15 at 16:45
  • This means your hosting setup is issuing high load. How many instances do you have in your app service? – Kerem Demirer Jan 19 '16 at 10:43
  • Our S2 App Service Plan has 1 Web App with 1 instance. – Guillaume Morin Jan 19 '16 at 18:50
  • @GuillaumeMorin, did you manage to find out more? I am having exactly the same behavior on my App Service hosting a SignalR hub. – Jacques Bosch Jun 03 '16 at 18:39
  • @JacquesBosch Nope. Been running like that for almost a year without issue though. I still believe those are signalR persistent connections. – Guillaume Morin Jul 18 '16 at 12:38
  • @GuillaumeMorin I agree. We are seeing a very clear correlation of the HTTPQ length and the number of connected clients. Here is my interaction with Azure Support about the same issue. https://social.msdn.microsoft.com/Forums/silverlight/en-US/05fc9c0f-d88a-4976-b52c-3468c00d87b3/high-http-queue-length-with-signalr-on-azuer-app-service?forum=windowsazurewebsitespreview – Jacques Bosch Jul 19 '16 at 14:53
  • Did you find any solutions to this problem. I have the exact same problem with all of my applications. Currently, I am disabling all SignalR related functionality to get rid of http queue length. – emre nevayeshirazi Oct 14 '16 at 22:23
  • No, haven't solved it, but I think it's been pretty much confirmed as expected reflection for SignalR persistant connections. The only frustrating part is that it is muddying the waters when you have non-SignalR sites running on the same plan. I.e. sites where you need to keep the HTTP queue short. – Jacques Bosch Nov 04 '16 at 19:39

3 Answers3

10

EDIT 11/12/2016

Based on some follow up information HTTP Queue Length will at some point soon here match up with expectations.

I got confirmation that after the fix, the HTTP Queue Length metric for Server Farm will be depicting the actual HttpQueueLength ( it is incorrectly showing the current Requests value at this time) for the Server Farm (App Service Plan). This fix will be rolled out in the coming weeks across all regions .

In talking with Azure Support HTTP Queue Length is actually a confusing name and maps to ''W3SVC_W3WP', 'Active Requests', '_Total'.

Here's the full response

Hello Shane, You are correct, we had noticed an issue with http queue metric , where we found out that in Auto-Scaling/Alert a Web App based on this metric was misleading as this metric was calculating the total active requests and not the queued requests and we had a discussion with our Product team about this metric and it appears that there is a little bit of ambiguity in the name of the metric HTTP Queue. In reality, the counter does we show on the portal corresponds to "W3SVC_W3WP", "Active Requests", "_Total". In other words, this counter does not represent a queue but rather requests that are running at any current time. We have also verified this by looking into our Product source code. We understand how the name of the counter could lead to confusion and we have submitted a request to our Product team to rename it. The Product team is concerned that renaming might break existing alerts customers might have but they are considering adding another metric named “Active Requests” which looks at the same value and later removing “HTTP Queue Length”. Keeping this in mind, our suggestion was to use any other metric (for example: CPU/Memory) to configure Auto-Scale/Alert rules instead of using Http Request Queue.

I had a discussion with our Product team just a few days back again and we are currently in the works on having this updated to represent that the counter value is for “Requests” and not the true Http Queue Length.

Though this distinction will soon be irrelevant as they also had this to say

Hi Shane, I was just testing this out on my end and I see that the portal has been updated at the Site level to show this metric as “Requests” . I think we are still waiting for this change to be pushed out to the Server Farm settings.

If I select Alert . In the “Resource” drop down select “sites” to select your site, then the metrics will list “Requests” , which will be the current requests.

So just something to be aware of that in its current state this metric does not indicate that you have requests backing up that the system can't handle.

Shane Neuville
  • 2,127
  • 14
  • 19
8

Http Queue Length: Count of pending HTTP operations. If your application is receiving more requests than the web server can handle, this could be your gap. That means there is a fallout of requests, and your current configuration is not enough to support the load. Use portal.azure.com the default for HTTP Queue Length starts from 1

enter image description here

I think you are using SignalR for Sockets, while sockets are maintaining the connection with your webserver, HTTP Queue Length is the count of web requests which were queued by the Azure just because it couldn't process any more, so yes may be, but not sure unless we analyze further.

Brij Raj Singh - MSFT
  • 4,903
  • 7
  • 36
  • 55
  • Cpu avg=10% and Memory avg=40%, so it doesn't seem resources are overloaded – Guillaume Morin Nov 23 '15 at 16:29
  • CPU and Memory is not the only metrics causes long request. Some other depenency like sql calls can increase response time and http queue. You can increase the instance count of AppService. I'm pretty sure that the queue will decrease. – Kerem Demirer Feb 15 '16 at 20:33
1

If requests end into the HTTP queue, it means that there is no thread available to process the requests. As Kerem suggests, it could be that there are some long I/O calls that put the threads into wait state (unable to serve requests). This is highl likely as CPU and Memory are low.

You could add Diagnostic.Trace on enter/exist methods and check the application log how long it takes to get in and out from those methods

Another way, attach with Visual Studio remotly and see what threads are doing. How many are waiting?

And another approach would be to take a memory dump (from kudu site perhaps). Then check what threads are doing: In WinDbg: !CLRStack -a *~ KB

You can also open the .dmp file in Visual Studio, select unmanaged, and check the threds. If all/nearly all the threads are in WaitFor***Object than they are waiting for IO completition. In this case, it's true that increasing instancing will solve/decrease the issue, but it would be better to change the methods to use the Async version for IO.

hth, Aldo

aljj
  • 106
  • 5
  • Thanks Aldo. Would you know if SignalR could cause this ? Ie, All clients have one or more 'long running' SignalR connection with the server. We have ~75 concurrent users, that may have multiple browser tabs opened. – Guillaume Morin Apr 08 '16 at 12:59
  • I am not familiar with SignalR. But you should focus on what happens when you serve a connection (or request). It looks like you are doing some I/O calls to a DB or something in a synchrounous way and the thread that does the call is put in WAIT waiting the response and unable to serve other requests. Under haevy load all threads ends in WAIT so other reqeusts get queued. You should change that I/O requests to async/await which would avoid threads ending WAIT. CLR via C# Jeffrey Richter has a good section on async and IO bound operations hth, aldo – aljj Apr 15 '16 at 14:27