How to implement an HTTP GET for large data avoiding Request Timeout error

Question

I`m developing WebAPI to transfer data between 2 systems. Simplifying I have to implement an HTTP GET method to retrieve large data. Same problem for a POST method to push data.

How can I do to avoid timeout error due to long processing?

Any help will be appreciated :)

Thanks

Send it in chunks/Use ranged requests/Support resuming/Use BitTorrent instead — Caius Jard, Feb 04 '21 at 19:26
@CaiusJard Thank you. I've already using chunks. Can you link me a sample using range requests and support resuming? What are you referring to with bitTorrent? — Andrea86, Feb 08 '21 at 15:45
Range is where you set the Range header e.g. if you already have 10241 bytes you can ask the server for `Range:10242-` and if it supports it it will send those bytes onwards (so not repeating stuf you already have) https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range — Caius Jard, Feb 08 '21 at 15:48
For upload similarly you can use HEAD to discover the size of file already on the server and then just post bytes the server doesn't have. For what is BitTorrent, google it? It's an efficient block based reliable transport system — Caius Jard, Feb 08 '21 at 15:48

Andrea86 · Answer 1 · 2021-02-13T19:01:33.837

For resuming problem I used Polly See below:

  public class RetryHandler : DelegatingHandler
  {
    // Strongly consider limiting the number of retries - "retry forever" is
    // probably not the most user friendly way you could respond to "the
    // network cable got pulled out."
    private const int MaxRetries = 3;

    public RetryHandler(HttpMessageHandler innerHandler)
        : base(innerHandler)
    { }

    // It doesn't work with Transient timeouts
    ////protected override async Task<HttpResponseMessage> SendAsync(
    ////    HttpRequestMessage request,
    ////    CancellationToken cancellationToken)
    ////{
    ////  HttpResponseMessage response = null;
    ////  for (int i = 0; i < MaxRetries; i++)
    ////  {
    ////    response = await base.SendAsync(request, cancellationToken);
    ////    if (response.IsSuccessStatusCode)
    ////    {
    ////      return response;
    ////    }
    ////  }

    ////  return response;
    ////}
    ///

    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken) =>
        Policy
            .Handle<HttpRequestException>()
            .Or<TaskCanceledException>()
            .OrResult<HttpResponseMessage>(x => !x.IsSuccessStatusCode)
            .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)))
            .ExecuteAsync(() => base.SendAsync(request, cancellationToken));
  }

I pass RetryHandler into HttpClient object:

this.Client = new HttpClient(new RetryHandler(new HttpClientHandler()))
      {
        BaseAddress = new Uri(uriText)
      };

How to implement an HTTP GET for large data avoiding Request Timeout error

1 Answers1