We are using Bull Queue delayed job feature to create timer for each user to do some specific tasks.
Our use case requires to create one job per user. Running at SCALE, we believe that vertically scaling a single REDIS machine will pose us a challenge. So we are experimenting on running Bull with REDIS running in cluster mode.
From documentation: To make bull compatible with Redis cluster, use a queue prefix inside brackets. A key that has a substring inside brackets will use that substring to determine in which HASH SLOT the key will be placed.
const queue = new Queue('cluster', {
prefix: '{myprefix}'
});
I created a queue with prefix: and when I add job's to this queue, following keys are created:
"{queueOne}:queueOne:1"
"{queueOne}:queueOne:2"
"{queueOne}:queueOne:id"
"{queueOne}:queueOne:delayed"
"{queueOne}:queueOne:5"
"{queueOne}:queueOne:4"
Looking at the keys, it apparent that all the job's will go to the same hash slot and hence to the same REDIS node. which is not desired solution for us.
So is there a way to distribute job's across nodes in a single queue, or do we have to implement a custom logic to solve our problem?
