Client-side request rate-limiting

Question

I'm designing a .NET client application for an external API. It's going to have two main responsibilities:

• Synchronization - making a batch of requests to API and saving responses to my database periodically.
• Client - a pass-through for requests to API from users of my client.

Service's documentation specifies following rules on maximum number of requests that can be issued in given period of time:

During a day:

• Maximum of 6000 requests per hour (~1.67 per second)
• Maximum of 120 requests per minute (2 per second)
• Maximum of 3 requests per second

At night:

• Maximum of 8000 requests per hour (~2.23 per second)
• Maximum of 150 requests per minute (2.5 per second)
• Maximum of 3 requests per second

Exceeding these limits won't result in immediate lockdown - no exception will be thrown. But provider can get annoyed, contact us and then ban us from using his service. So I need to have some request delaying mechanism in place to prevent that. Here's how I see it:

public async Task MyMethod(Request request)
{
  await _rateLimter.WaitForNextRequest(); // awaitable Task with calculated Delay
  await _api.DoAsync(request);
  _rateLimiter.AppendRequestCounters();
}

Safest and simpliest option would be to respect the lowest rate limit only, that is of max 3 requests per 2 seconds. But because of "Synchronization" responsibility, there is a need to use as much of these limits as possible.

So next option would be to to add a delay based on current request count. I've tried to do something on my own and I also have used RateLimiter by David Desmaisons, and it would've been fine, but here's a problem:

Assuming there will be 3 requests per second sent by my client to the API at day, we're going to see:

• A 20 second delay every 120th request
• A ~15 minute delay every 6000th request

This would've been acceptable if my application was only about "Synchronization", but "Client" requests can't wait that long.

I've searched the Web, and I've read about token/leaky bucket and sliding window algorithms, but I couldn't translate them to my case and .NET, since they mainly cover the rejecting of requests that exceed a limit. I've found this repo and that repo, but they are both only service-side solutions.

QoS-like spliting of rates, so that "Synchronization" would have the slower, and "Client" the faster rate, is not an option.

Assuming that current request rates will be measured, how to calculate the delay for next request so that it could be adaptive to current situation, respect all maximum rates and wouldn't be longer than 5 seconds? Something like gradually slowing down when approaching a limit.

Answer 1

This is achievable by using the Library you linked on GitHub. We need to use a composed TimeLimiter made out of 3 CountByIntervalAwaitableConstraint like so:

var hourConstraint = new CountByIntervalAwaitableConstraint(6000, TimeSpan.FromHours(1));
var minuteConstraint = new CountByIntervalAwaitableConstraint(120, TimeSpan.FromMinutes(1))
var secondConstraint = new CountByIntervalAwaitableConstraint(3, TimeSpan.FromSeconds(1));

var timeLimiter = TimeLimiter.Compose(hourConstraint, minuteConstraint, secondConstraint);

We can test to see if this works by doing this:

for (int i = 0; i < 1000; i++)
{
    await timeLimiter;
    Console.WriteLine($"Iteration {i} at {DateTime.Now:T}");
}

This will run 3 times every second until we reach 120 iterations (iteration 119) and then wait until the minute is over and the continue running 3 times every second. We can also (again using the Library) easily use the TimeLimiter with a HTTP Client by using the AsDelegatingHandler() extension method provided like so:

var handler = TimeLimiter.Compose(hourConstraint, minuteConstraint, secondConstraint);
var client = new HttpClient(handler);

We can also use CancellationTokens, but as far as I can tell not at the same time as also using it as the handler for the HttpClient. Here is how you can use it with a HttpClientanyways:

var timeLimiter = TimeLimiter.Compose(hourConstraint, minuteConstraint, secondConstraint);
var client = new HttpClient();

for (int i = 0; i < 100; i++)
{
    await composed.Enqueue(async () =>
    {
        var client = new HttpClient();
        var response = await client.GetAsync("https://hacker-news.firebaseio.com/v0/item/8863.json?print=pretty");
        if (response.IsSuccessStatusCode)
            Console.WriteLine(await response.Content.ReadAsStringAsync());
        else
            Console.WriteLine($"Error code {response.StatusCode} reason: {response.ReasonPhrase}");
    }, new CancellationTokenSource(TimeSpan.FromSeconds(10)).Token);
}

Edit to address OPs question more:

If you want to make sure a User can send a request without having to wait for the limit to be over with, we would need to dedicate a certain amount of request every second/ minute/ hour to our user. So we need a new TimeLimiter for this and also adjust our API TimeLimiter. Here are the two new ones:

var apiHourConstraint = new CountByIntervalAwaitableConstraint(5500, TimeSpan.FromHours(1));
var apiMinuteConstraint = new CountByIntervalAwaitableConstraint(100, TimeSpan.FromMinutes(1));
var apiSecondConstraint = new CountByIntervalAwaitableConstraint(2, TimeSpan.FromSeconds(1));

// TimeLimiter for calls automatically to the API
var apiTimeLimiter = TimeLimiter.Compose(apiHourConstraint, apiMinuteConstraint, apiSecondConstraint);

var userHourConstraint = new CountByIntervalAwaitableConstraint(500, TimeSpan.FromHours(1));
var userMinuteConstraint = new CountByIntervalAwaitableConstraint(20, TimeSpan.FromMinutes(1));
var userSecondConstraint = new CountByIntervalAwaitableConstraint(1, TimeSpan.FromSeconds(1));

// TimeLimiter for calls made manually by a user to the API
var userTimeLimiter = TimeLimiter.Compose(userHourConstraint, userMinuteConstraint, userSecondConstraint);

You can play around with the numbers to suit your need.

Now to use it:
I saw you're using a central Method to execute your Requests, this makes it easier. I'll just add an optional boolean parameter that determines if it's an automatically executed request or one made from a user. (You could replace this parameter with an Enum if you want more than just automatic and manual requests)

public static async Task DoRequest(Request request, bool manual = false)
{
    TimeLimiter limiter;
    if (manual)
        limiter = TimeLimiterManager.UserLimiter;
    else
        limiter = TimeLimiterManager.ApiLimiter;

    await limiter;
    _api.DoAsync(request);
}

static class TimeLimiterManager
{
    public static TimeLimiter ApiLimiter { get; }

    public static TimeLimiter UserLimiter { get; }

    static TimeLimiterManager()
    {
        var apiHourConstraint = new CountByIntervalAwaitableConstraint(5500, TimeSpan.FromHours(1));
        var apiMinuteConstraint = new CountByIntervalAwaitableConstraint(100, TimeSpan.FromMinutes(1));
        var apiSecondConstraint = new CountByIntervalAwaitableConstraint(2, TimeSpan.FromSeconds(1));

        // TimeLimiter to control access to the API for automatically executed requests
        ApiLimiter = TimeLimiter.Compose(apiHourConstraint, apiMinuteConstraint, apiSecondConstraint);

        var userHourConstraint = new CountByIntervalAwaitableConstraint(500, TimeSpan.FromHours(1));
        var userMinuteConstraint = new CountByIntervalAwaitableConstraint(20, TimeSpan.FromMinutes(1));
        var userSecondConstraint = new CountByIntervalAwaitableConstraint(1, TimeSpan.FromSeconds(1));

        // TimeLimiter to control access to the API for manually executed requests
        UserLimiter = TimeLimiter.Compose(userHourConstraint, userMinuteConstraint, userSecondConstraint);
    }
}

This isn't perfect, as when the user doesn't execute 20 API calls every minute but your automated system needs to execute more than 100 every minute it will have to wait.

And regarding day/ night differences: You can use 2 backing fields for the Api/UserLimiter and return the appropriate ones in the { get {...} } of the property