3

I have created a dotnet core web api using Asp.Net Core Web Application (.Net Core) template which gives us api/values controller and on the post i am sending a message to azure storage queue using Windows Azure Storage library v8.0. https://www.nuget.org/packages/WindowsAzure.Storage/8.0.0

Currently when i do a single request, the queue takes about an avg. 140ms to complete the method AddMessageAsync() but when i do a load test for 200 request per second the same method takes an average 800ms to complete. According to azure storage queue it should be able to handle 2000 request per sec but i am not able to get 200 request per second.

I would appreciate if someone to provide an some information as why web application api not performing as expected.

Please see below my code example

Startup.cs - ConfigureServices()

// Add QueueAccessLayer.
services.AddSingleton<IQueueAccessLayer, QueueAccessLayer>();

Emailcontroller.cs

[Route("api/[controller]")]
public class EmailController : Controller
{
    private IQueueAccessLayer _queue;

    public EmailController(IQueueAccessLayer queue)
    {
        _queue = queue;
    }

    // POST api/values
    [HttpPost]
    public async Task<IActionResult> Post([FromBody]string value)
    {
        var emailMessage = "Message Id - " + Guid.NewGuid();
        await _queue.SendMessage(emailMessage);
        return new EmptyResult();
    }
}

QueueAccessLayer.cs

public class QueueAccessLayer : IQueueAccessLayer
{
    private CloudQueueClient _queueClient;         
    private CloudStorageAccount _storageAccount;

    private CloudQueue _emailQueue;
    private ILogger<QueueAccessLayer> _logger;


    public QueueAccessLayer(ILogger<QueueAccessLayer> logger)
    {
        _storageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=test1;AccountKey=#####;");
        _queueClient = _storageAccount.CreateCloudQueueClient();
        _emailQueue = _queueClient.GetQueueReference("emailqueue");
        _emailQueue.CreateIfNotExistsAsync().Wait();

        _logger = logger;
    }

    public async Task<bool> SendMessage(string msg)
    {
        Stopwatch watch = new Stopwatch();
        watch.Start();
        CloudQueueMessage message = new CloudQueueMessage(msg);
        await _emailQueue.AddMessageAsync(message);
        watch.Stop();

        _logger.LogInformation(msg + " - " + watch.ElapsedMilliseconds + "ms");
        return true;
    }
}

public interface IQueueAccessLayer
{
    Task<bool> SendMessage(string msg);
}
user1754675
  • 887
  • 13
  • 32
  • RPS also limited by WebApp, please have a try to [change tier or scale out of App service plan](https://learn.microsoft.com/en-us/azure/app-service/azure-web-sites-web-hosting-plans-in-depth-overview) to check whether there is any improvement for RPS? – Tom Sun - MSFT Jan 27 '17 at 06:32
  • Currently my web app is running locally in a docker container. – user1754675 Jan 27 '17 at 06:43
  • I understand but if I comment this line await _queue.SendMessage(emailMessage); I get around 1814 request per second with 72ms avg response. With the above I get 152 rps with 600ms avg response. – user1754675 Jan 27 '17 at 07:01
  • Azure storage or local storage emulator? If you are using Azure storage, please have a try to test with [local storage emulator](https://learn.microsoft.com/en-us/azure/storage/storage-use-emulator). – Tom Sun - MSFT Jan 27 '17 at 07:09
  • I have tested with local storage and it works fine getting 200 rps with 26ms Avg response but if I change to cloud queue I am seeing the problem. – user1754675 Jan 27 '17 at 13:23
  • Have you tried disabling Nagle algorithm and retesting? (chances are, in high-load scenarios, it won't have an effect, but there's still a chance). I posted details [here](http://stackoverflow.com/a/19524403/272109). – David Makogon Jan 27 '17 at 15:08
  • I will give a try later today but would I not see similar behaviour against my local azure emulator. – user1754675 Jan 27 '17 at 15:12
  • David - how can I disable Nagle algorithm in dot core project ? – user1754675 Jan 27 '17 at 18:36
  • @user1754675 - ah - just realized you were talking about core. Looks like it hasn't been addressed, according to [this issue](https://github.com/dotnet/corefx/issues/10727). It still might be worth you benchmarking with non-core .net to compare performance. – David Makogon Jan 27 '17 at 20:33
  • It seems that network or some other things cause the performance lower. Is Azure cloud storage location the same as your location? If it is not that case , please have a try to use the some location to have a try. – Tom Sun - MSFT Jan 31 '17 at 08:37
  • After raising a support ticket with Microsoft, they asked me to turn off Nagle so to test the performance i had to convert to .Net 4.5.2 and deploy to azure web app closer to queues to remove latency. Here i found that it is using high CPU around 80% on heavy load where i managed to send about 800 hits/s on 2 instances (4-Core 7Gb each). Any suggestion to reduce CPU usage ? – user1754675 Feb 02 '17 at 09:02

1 Answers1

1

Make sure to set the following on application startup:

  1. ServicePointManager.Expect100Continue = false;
  2. ServicePointManager.UseNagleAlgorithm = false;

Expect100Continue set to false - will reduce roundtrips to server side when sending requests.

UseNagleAlgorithm set to false - will turn off Nagle optimization and dramatically improve performance.

There's a great blog that explains this: Nagle’s Algorithm is Not Friendly towards Small Requests .

Nagling is a TCP optimization on the sender and it is designed to reduce network congestion by coalescing small send requests into larger TCP segments... This is achieved by holding back small segments either until TCP has enough data to transmit a full sized segment or until all outstanding data has been acknowledged by the receiver... The test is run as a worker role accessing our storage account in the same geo location. It inserts messages which are 480 bytes in length. The results show that the average latency improves by more than 600% with Nagling turned off

yonisha
  • 2,956
  • 2
  • 25
  • 32