1

I'm using curl to request a free external 3rd party API in my PHP application. When there are too many requests at a time, it goes down and shows 504 Gateway time-out. Is there any option, when a request has no response in more than 5s then it would stop waiting for API response & go for the next statements?

As I know while adding an external API in an application, becomes a dependency. Need your attention & guidance.

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
R G
  • 11
  • 1

2 Answers2

0

What happens is that the 3rd party web server triggers “self-defense” by returning 504 error, when can’t handle certain amount of requests per minute. To avoid that you need to honor their quality of service (QoS) by throttling down the number of requests right before reaching the limit or after you get the error. You can do that by:

  1. Find how many request per minute the 3rd party site can handle either by a) looking at their docs or asking, b) checking the result from the API how many requests you are allowed to do in a minute or by c) experimenting and measuring it.
  2. Use some kind of queue that all requests for that API will be added to to it and a code that will handle messages once there is message in the queue, then call the API, then remove from the queue or move to failed_queue. For queue you could use external library or create a simple table that saves messages as records. Code checking and handling the queue could be executed on a corn job.
  3. Count how many requests you send so far and the last time since when the previous request was send. Check, if you if the requests are less than the limit-1. If is less execute the API requests, if it’s more or 504 was received than wait 1 minute, e.g.’sleep(60)’, after that send a request again.

It’s better to avoid 504 before it happens, because the API website could return it for longer time, than the time needed to prevent it.

Svetlozar Stoyanov
  • 146
  • 1
  • 5
0

To expound on what Svetlozar said, if you get a timeout because the service was unavailable, due to an exorbitant number of request, and you skip to the next curl request, you are just sending more requests while the service is unavailable. This could lead to an extended down time as well as you not getting the results.

You could and should do as Svetlozar suggested and throttle your requests. Find out what the threshold is and stay below it.

Depending on the API and your exact purpose (specifically how fresh your data has to be), you may also consider caching the results and pulling from the cache before pulling directly from the API. You then just have to expire your results after a certain amount of time to make sure your data is not too stale. This would help prevent you from making so many rapid-fire requests.

I did the caching for a caller ID API. Everytime a call came in the phone server would contact my internal server and request the CID information. If my server had cached the data, it provided it, if it didn't have a cached version then it would send the request to the API. Any data more than one month old would get pruned. This cut down on the number of requests that actually had to be made to the API significantly because of the repeat callers.

James McCaughey
  • 3
  • 2