3

We are experiencing an issue with HTTP requests where we are seeing that every 6 minutes a HTTP request is taking up to 162ms, on average these requests are taking 7ms. The metrics graph below shows a brief run of a K6 script calling our controller at a rate of 36 requests/minute. Im running K6 locally and both services in question are an AWS ECS instances. The slow response pattern persists for as long as I leave K6 running at this rate. The more you increase the rate at which K6 sends requests the less noticeable the spikes in the requestduration metric. I have metrics around the API controller which I am calling and they are showing an average time taken of 2ms so this is not the issue.

requestduration graph

I initially thought this might be to do with the PooledConnectionIdleTimeout, but the default value for this is 1 minute in .NET 6 and later versions. We are using .Net 7. Not seeing anything on our infrastructure which would cause these spikes either.

My question is has anyone seen similar behaviour making HTTP requests, is it a possible issue with the code below?

We are adding our HTTP Client to our IServiceCollection as follows:

_builder.Services.AddHttpClient<IHttpClient, HttpClient>();

Our HTTP Client is as follows

public class HttpClient : IHttpClient
{
    private readonly Metric.Client.IRecorder c_metrics;
    private readonly System.Net.Http.HttpClient c_httpClient;
    
    public HttpClient(
        Metric.Client.IRecorder metrics,
        System.Net.Http.HttpClient httpClient)
    {
        this.c_httpClient = httpClient;
    }

    public async Task<HttpResponseMessage> GetAsync(
        string requestUrl,
        string correlationId)
    {
        using (var _request = new HttpRequestMessage(HttpMethod.Get, requestUrl))
        {
            _request.Headers.Add("X-CorrelationId", correlationId);
            
            using var _timer = this.c_metrics.StartTimer("requestduration");
            return await this.c_httpClient.SendAsync(_request);
        }
    }
}

Our controller which we get our HttpClient from our DI container

public Controller(
    IHttpClient httpClient)
{
    this.c_httpClient = httpClient;
}

public async Task<IActionResult> GetAsync(
    [FromQuery] Request request)
{
    var _privateResponse = await this.c_httpClient.GetAsync(_requestUrl, _correlationId);

    if (_privateResponse.StatusCode != HttpStatusCode.OK)
    {
        return StatusCode(500, new Error(Guid.Parse(_correlationId), "Internal server error"));
    }

    var _privateResponseContent = await _privateResponse.Content.ReadAsStringAsync();

    return Ok(JsonSerializer.Deserialize<Response>(_privateResponseContent));
}
Alan Mulligan
  • 1,107
  • 2
  • 16
  • 35
  • What happens if your K6 script calls the API that your controller is calling at the rate you call your API? Is your API the bottleneck or the other API? – Neil May 22 '23 at 15:21
  • Might be GC, have you set GC to server-mode? https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/workstation-server-gc – Charlieface May 22 '23 at 15:47
  • 2
    The issue could be another process on the machine running or the network. You could use a sniffer like wireshark or fiddler to isolate if issue is with Network. – jdweng May 22 '23 at 15:47
  • how big is the string in `_privateResponseContent`? It's generally a bad idea to read in as string, then deserialize said string. You should let the deserailizer do all that via stream. Also, this is unrelated, but you should use `_privateResponse.IsSuccessStatusCode`. there are many more successful status' other than `200 OK`. – Andy May 22 '23 at 19:41
  • 1
    You are calling an external dependency. It could be that external dependency does it, not your code. You should eliminate that assumption first. – Vlad DX May 22 '23 at 19:58

1 Answers1

1

Have you tried setting the PooledConnectionIdleTimeout & PooledConnectionLifetime? I suspect it is the former one you need to modify for low traffic.

using var socketHandler = new SocketsHttpHandler()
{
    PooledConnectionIdleTimeout = TimeSpan.FromMinutes(XXX),
    PooledConnectionLifetime = TimeSpan.FromMinutes(YYY),
};

var httpClient = new HttpClient(socketHandler);
Jmt
  • 11
  • 1