3

Suppose I have a simple HTTP client with a naïve method like this:

def httpGet(url: String)(implicit ec: ExecutionContext): Future[String] = Future {
  io.Source.fromURL(url).mkString
}

Suppose also I call it against a server with a rate and concurrency limit. I think these limits should be implemented in ExecutionContext.

The concurrency limit is just the number of threads of the java.util.concurrent.Executor backing some ExecutionContext. The rate limiting should be part of the ExecutionContext too. So we can write a new class extending ExecutionContext to implement the rate and concurrency limits and use an instance of this class to invoke httpGet.

class ExecutionContextWithRateAndConcurrencyLimits(
  numThreads: Int, // for concurrency limit
  rate: Rate       // requests per time unit
) extends ExecutionContex { ... }


val ec = new ExecutionContextWithRateAndConcurrencyLimits(
  numThreads = 100,
  rate = Rate(1000, 1.sec)
)

val fut = httpGet(url)(ec)

Would you agree that rate and concurrency limits should be implemented in ExcecutionContext? Do you know any implementation like that ?

Michael
  • 41,026
  • 70
  • 193
  • 341
  • 1
    `ExecutionConext` is way more general than just for doing HTTP requests, even in that simple case it does more than just doing the petition. Thus, the **EC** should not take care of that, rather you may want to create a custom HTTP client that manages and abstracts all those concepts from you; there are probably libraries that already do that. – Luis Miguel Mejía Suárez May 07 '21 at 14:48
  • 1
    Thanks. I agree that EC is more than just HTTP. However rate and concurrency limits are more general than HTTP too. They are part of _execution logic_. That's why I argue they belong to EC. – Michael May 07 '21 at 14:56
  • Also I guess it's easier to implement these limits in `ThreadPoolExecutor`. Concurrency limit is just the number of threads. Rate limit is just some logic on top of the the blocking queue. Then we can implement an EC on top of this `ThreadPoolExecutor` and that's it. – Michael May 07 '21 at 15:16
  • _"Concurrency limit is just the number of threads"_ ham not necessarily, you can have concurrency with a single thread, take a look to **JS**. _"Rate limit is just some logic on top of the blocking queue"_ sure, but you can also do it without directly interacting with that. - The biggest problem with your approach is that the **EC** manages not only the HTTP petitions, but also the callbacks and further processing; which you do not want to modify. So, you would need to swap **EC** before and after the petition _(which would be pretty similar that just having a wrapper over the client)_. – Luis Miguel Mejía Suárez May 07 '21 at 15:28
  • Thanks for the explanation. I see your point. How would you implement rate limiting on top of the client ? – Michael May 07 '21 at 15:34
  • However, you are right that this logic is also more general than just HTTP clients. - I am not sure how would I do it with **Futures**, I do not use them for anything real since a long time ago. In the **cats-effect** ecosystem I use libraries like [this](https://github.com/SystemFw/upperbound) and [this](https://github.com/cb372/cats-retry) for this kind of things. I guess the concurrency limit is indeed well solved using a custom **EC** backed by a `Executors.newFixedThreadPool` about the rate limiter you would need some kind of scheduler and lazy futures, maybe **AkkaStreams** provides that – Luis Miguel Mejía Suárez May 07 '21 at 16:28
  • 1
    Oh, I am glad we can agree :)) I will take a look at the libraries you suggested. However I'd prefer a "vanilla" solution without 3d-party deps. – Michael May 07 '21 at 16:53
  • What is your desired behavior? With your example you should be able to to at most 1000 requests per 1s, right? – Krzysztof Atłasik May 07 '21 at 19:36
  • Yes, I want to send 1000 requests per sec to the server. If the client generates more requests per sec. (and the internal request queue is full) then `httpGet` should return a failed future. – Michael May 07 '21 at 19:48

0 Answers0