1

We have ASP.Net Core 2.1 Web API hosted in AppService (S1) that talks to Azure SQL DB (S1-20DTUs). Both are in same region. During load testing we found that some API instances are taking too much time to return the result.

We tried to troubleshoot the performance issue and below are our observations.

  1. API responds within 0.5 secs most of the time.
  2. API methods are all async methods.
  3. Sometimes it takes around 50 secs to over a minute.
  4. CPU & Memory utilization are below 60%
  5. Database has 20 DTU capacity, out of which 6 DTUs are used during load testing.
  6. In the below example snapshot from Application Insights, we see total duration of the request was 27.4 secs. But the database dependency duration was just 97ms. There is no activity till the database was called. Please refer below example.

Can someone please help me to understand what was happening in this 27 secs of wait time. What could be the reason for this?

enter image description here

Praveena M
  • 522
  • 4
  • 10

2 Answers2

1
  1. I would recommend checking the Application Map on Application Insights resource as shown below to double check the dependencies.

enter image description here

  1. Verify the CPU and Memory metrics by going to the "Diagnose and solve problems" link on App service as shown below and run the Availability and Performance report to find out if there were any issues during your load testing.

enter image description here

  1. Use Async methods on your API to maximize the CPU usage. It may be that the worker process threads are hitting the limits and your app is the bottleneck. You should get some insights when you run the report mentioned in point 2 above.
rohit
  • 666
  • 8
  • 24
  • I have already seen the Application Map, there is no other dependency for this particular api call. However the application has several other api method does outbound http calls. Yes we have all the api methods as an async. Also I looked into Diagnostics and Solve problems, that listed out 502.3 errors. – Praveena M May 06 '20 at 14:17
  • OK. So, 503.2 means service unavailable which was probably due to the hammering of load testing and hence caused the delay? – rohit May 06 '20 at 14:44
0

The S1 tier will support no more than 900 concurrent session. If you request per second (RPS rate) during the load test is very high you may face issues.

Also S3 and above are recommended for intensive workloads. Checking if all the connections are closed properly also helps

You can find details about different pricing tiers and their capabilities in the below link https://learn.microsoft.com/en-us/azure/sql-database/sql-database-dtu-resource-limits-single-databases

Guru Pasupathy
  • 440
  • 1
  • 4
  • 13
  • do you think the database is the bottleneck? Database query was started at around 27th second and returned in just 97 ms. I am kind of worried, what appservice was doing till 27 seconds. How to find more about this? – Praveena M May 06 '20 at 14:13
  • 1
    Few things I can think of 1. If you haven't enabled auto scaling, you can give it a try. During your load test, auto scaling is triggered AND you don't face any issue, it can indicate bottleneck with the pricing tiers / number of instances 2. Similarly, you may try pushing the Db to S3 and see how it goes 3. You had mentioned that you are calling some external APIs. Can you check the how these API are performing under stress? Maybe you can get some metrics. Even though you are calling them asyn, could you check if there is any code block that effectives waits for these to return? – Guru Pasupathy May 06 '20 at 15:19