0

I'm thinking about making a worker script to handle async tasks on my server, using a framework such as ReactPHP, Amp or Swoole that would be running permanently as a service (I haven't made my choice between these frameworks yet, so solutions involving any of these are helpful).

My web endpoints would still be managed by Apache + PHP-FPM as normal, and I want them to be able to send messages to the permanently running script to make it aware that an async job is ready to be processed ASAP.

Pseudo-code from a web endpoint:

$pdo->exec('INSERT INTO Jobs VALUES (...)');
$jobId = $pdo->lastInsertId();

notify_new_job_to_worker($jobId); // how?

How do you typically handle communication from PHP-FPM to the permanently running script in any of these frameworks? Do you set up a TCP / Unix Socket server and implement your own messaging protocol, or are there ready-made solutions to tackle this problem?

Note: In case you're wondering, I'm not planning to use a third-party message queue software, as I want async jobs to be stored as part of the database transaction (either the whole transaction is successful, including committing the pending job, or the whole transaction is discarded). This is my guarantee that no jobs will be lost. If, worst case scenario, the message cannot be sent to the running service, missed jobs may still be retrieved from the database at a later time.

BenMorel
  • 34,448
  • 50
  • 182
  • 322

2 Answers2

0

If your worker "runs permanently" as a service, it should provide some API to interact through. I use AmPHP in my project for async services, and my services implement HTTP/Websockets servers (using Amp libraries) as an API transport.

Edward Surov
  • 469
  • 4
  • 3
0

Hey ReactPHP core team member here. It totally depends on what your ReactPHP/Amp/Swoole process does. Looking at your example my suggestion would be to use a message broker/queue like RabbitMQ. That way the process can pic it up when it's ready for it and ack it when it's done. If anything happens with your process in the mean time and dies it will retry as long as it hasn't acked the message. You can also do a small HTTP API but that doesn't guarantee reprocessing of messages on fatal failures. Ultimately it all depends on your design, all 3 projects are a toolset to build your own architectures and systems, it's all up to you.

WyriHaximus
  • 1,000
  • 6
  • 8
  • I'm not planning to use RabbitMQ as explained at the end of my question: I need to atomically commit the job together with the database transaction. With RabbitMQ, I could get a committed transaction and a lost message, or the other way around. Hence my question of just sending some kind of push notification to ReactPHP! – BenMorel Nov 07 '19 at 15:59
  • Ah doh sorry missed that since it's tiny and all. What you could do is poll the table every second (or even every 0.1 second) and not notify ReactPHP. That way you purely trust on the database transaction and don't need the notification. And depending on your database you might be able to subscribe to table events and use that as notification. The thing is that any other notification system, sockets, http, udp, would have the same possible issues as you describe with rabbitmq. – WyriHaximus Nov 07 '19 at 19:12
  • I want to avoid polling as much as possible. My goal is to send a push notification to instantly process jobs, and if any notification is lost, I would only poll for jobs with *lost* notifications, like every minute or so. – BenMorel Nov 08 '19 at 09:00
  • Assuming your on PostgreSQL these two together should do the trick for you: https://citizen428.net/blog/asynchronous-notifications-in-postgres/ && https://github.com/voryx/PgAsync/#example---listennotify – WyriHaximus Nov 09 '19 at 20:16