BullMQ - Group/Job based completion

Question

I am using BullMQ, Redis and MySQL in a producer-consumer model to process jobs.

I have a producer that looks like this:

const jobOptions = {
    removeOnComplete: true, // remove job if complete
    delay: 60000,
    attempts: 3
};

const sasWorkerProducerService = ({redisConnection}) => {
    const queueLogId = async (jobId, logIds) => {
        for(const logId of logIds) {
            redisConnection.add({
                jobId: jobId,
                scraperPriceLogId: logId
            }, jobOptions);
        }
    }
    return {
        queueLogId
    }
}

module.exports = sasWorkerProducerService;

And I have a worker service that handles the jobs:

const Bull = require('bull');

const connectQueue = (name) => new Bull(name, {
    redis: {
        port: 6379, host: 'xxxx', password: 'xxxx'
    },
    settings: {
        maxStalledCount: 5
    }

})

module.exports = { connectQueue }

        const nameQueue = 'sas-worker'
        const cases = await connectQueue(nameQueue)

        const initJob = () => {
            console.info('job is working!');
            cases.process('__default__', 300, processJob);
            cases.on('failed', handlerFailure);
            cases.on('completed', handlerCompleted);
            cases.on('stalled', handlerStalled);
        }
        initJob()

Notice that the producer sends a jobId as part of the payload. This ID is an identifier that I have generated and stored in a MySQL database. A job represents a batch of items that need to be processed. I don't care about what order the jobs for a batch get completed in.

However, how can I determine once all of the jobs for a given jobId have been completed? I need to do some work after all the jobs have been processed.

I understand the nature of a a producer-consumer model is to do work on an item and forget about it, but how can I do some final, post-processing work for a job after all the items have indeed been processed?

BullMQ - Group/Job based completion

0 Answers0