I'm new to working with distributed queue systems, so I guess my question may be weird. But really hope to find a solution.
I have a Node.js microservice that uses Bull Queue. The service works with 2 queues: it subscribes on the firstQueue and produces jobs for the secondQueue when getting the message.
The return value of the firstQueue contains data for multiple jobs for the secondQueue. So I produce the jobs like this:
firstQueue.on('global:completed', async (jobId, returnValue) => {
const data = JSON.parse(returnValue);
for (const task of data) {
secondQueue.add(task, jobSettings);
}
});
And I have the process worker for the secondQueue in the same service:
const workerProcess = require('./workerProcess');
firstQueue.process(workerProcess);
It works well, but I want to process these secondQueue tasks in parallel. The microservice is running as Docker Service in Docker Swarm mode. So the first decision I made was to use replicas: N option for Docker service. But here is a problem: I want to have replicated process worker for secondQueue, but I'm also getting replicated firstQueue listener, hence I read the same message from firstQueue N times. Actually, I want to receive the message only once and just parallelize job processes.
I think, using of replicas option for Docker service cannot help me. And I have no idea how to achieve what I want. Are there some ways/patterns to do that? Or maybe I've chosen the wrong architecture for my microservice?
