I have to call third-party API in bulk(calling API more than 1000 times) and the third-party API has a rate limit of 10 request per second.
Here is the architecture to call the third-party API.
scheduler
- which will take data from the primary database and publish it into a message broker with the API rate limit.Message broker
- It has dynamic numbers of consumerInternal API
- which will process the data and invoke the third-party API.
Now, the Above architecture will help to handle the API rate limit, but the problem will arise when the third party will take time due to any reason(Network latency or etc..) and due to this it is possible that more messages can stay in the message broker and which will lead to API rate limit.
To overcome this I have decided to do inter-process communication in which when(#3) will get the 429
then it will tell Cron -Job (#1) to pause the publishing message for some amount of time and then continue after the time.
Which framework or library will help to achieve this?