0

I'm building an API using Laravel that has an endpoint /api/keyword. Clients can use that API to request statistics about given keywords (provided as a query parameter). Those statistics are coming from a 3rd party service that has uses throttling (max. 3 requests per minute) and only accepts 100 keywords at a time. Hence, I would like to reduce requests (to the 3rd party API) by potentially combining keywords coming from multiple requests (from the client).

I've implemented a job called FetchKeywordStatistics which is dispatched in the API controller like this:

FetchKeywordStatistics::dispatch($keywords); // $keywords being [Keyword, Keyword, ...]

Ideally, what I would like to achieve is when the queue is picking up a job, it looks at other queued jobs and optionally merge those jobs by combining the keywords and making a single request to the 3rd party service.

Is this somehow possible?

Niels Mouthaan
  • 1,010
  • 8
  • 19
  • why dont you do it as a scheduled task instead of jobs? that way if there is multiple keywords to use, you merge them in a task. – N69S Nov 24 '22 at 20:57
  • @N69S Thanks. You're referring to [scheduling a job](https://laravel.com/docs/9.x/scheduling#scheduling-queued-jobs)? Or a task that runs each minute to gather pending keywords (stored in the database from the controller) and process them (preferably) in one go? The latter sounds doable, but ideally, I would like the task to run as soon as possible (after a request comes in). Now there's a possibility I need to wait 59 seconds for the statistics to be requested. Preferably I just want a min. 20 seconds delay between the requests to the 3rd party service. – Niels Mouthaan Nov 24 '22 at 21:09
  • you can also do it with a job that "gather pending keywords (stored in the database from the controller) and process them (preferably) in one go". You just need to delay the job some 20 seconds. – N69S Nov 24 '22 at 22:24
  • @N69S I agree it can be done when jobs are stored in a MySQL database. But if they are stored in REDIS, SQS or others. It'll be much more complex and time-consuming. I'd go with your first solution: store the "task" list in DB or temporary cache, run a cronjob every 10-20s that pulls the "task" list, and execute third-party API calls. – Mtxz Nov 25 '22 at 00:11
  • @Mtxz cron job can run as fast as once per minutes. You can't configure a cron task to run every 10 seconds. – N69S Nov 25 '22 at 08:09
  • @N69S indeed you are right. But there are some tweaks to do it: https://stackoverflow.com/questions/30295868/how-to-setup-cron-job-to-run-every-10-seconds-in-linux – Mtxz Nov 25 '22 at 16:44
  • Thank all. Implemented a custom keyword queue and this works okayish. There's a job that picks keywords from the queue to request statistics from the 3rd party service. I'm dispatching instances of this job from the controller to ensure statistics are gathered as soon as possible (instead of a scheduled cron job). I tried [rate limiting](https://laravel.com/docs/9.x/queues#rate-limiting) the job to enforce the 3rd party service is accessed max. 3 times in a minute, but instead of delaying the job, the job fails. Will look into this further . – Niels Mouthaan Nov 25 '22 at 20:17

0 Answers0