I'm currently working on a competitive programming website using Node.js for the backend. I've set up a Bull task queue to manage code submissions, and things are going well. However, its currently a monolith and I wanted to learn Go so I have decided to write a microservice in Go to handle these submissions by having multiple of these workers.
Here's what I'm thinking:
- I'll use a central
beanstalkdqueue to receive submission tasks from the Node.js API. - The plan is to create worker nodes in Go that can process these tasks by subscribing to the queue.
Now, here's the challenge: figuring out how to send the results back from these Go workers to the original HTTP request context. With Bull in Node.js, I could just use something like
await job.finished()
but since it's a shared queue, there's nothing like this in fivebeans which is the Node.js library to interact with beanstalkd
One idea I'm considering is introducing another queue (or tubes, if I'm understanding correctly), but I'm concerned about the Node.js API constantly polling this queue. Given JavaScript's performance characteristics and since this would be over the network, this might not be the most efficient approach.
I'd really appreciate any suggestions or alternative ideas you might have. I'm open to exploring options beyond relying solely on the beanstalkd queue and I am looking for suggestions related to using particularly these two:
- gRPC
- RabbitMQ
