bull queue block job while got job on different function

Question

I need an approach to block worker to process a job while I called getJob on different function. I've looked around but couldn't find a solution for that.

I have following setup. In nodeJS with express, I have worker node.

Job created with delayed state.
Job is being accessed in different function

async function jobReader(id) {
   const job = await queue.getJob(id);
   /* do some stuff */
   await job.remove();
}

Worker node that independently processes jobs. Job will be only processed if the delayed time is finishes.

queue.process(async (job) => {
   /* do some stuff */
})

queue.getJob(id) doesn't block the worker to process the job. So there's race on worker processing the job and jobReader processing the job. I am writing some result to DB according to job status. So the race condition is not acceptable.

Apparently, getJob is not blocking the worker to process the job. Is there any way to lock or block to worker work on the job, if the job is read by some other function with getJob function.

Any help or documentation will be appreciated.

Thanks

I guess you have to rethink your logic. IF you want the jobreader, to always be first, then pass the job to `JobReader` and let `Jobreader `pass it to `queue`? Can you provide more code and a better, more practical example? I don't quiet get your question — Silvan Bregy, Jul 09 '21 at 09:42
First, user INITIATE the transaction, with next request, he can do (APPROVE,FAIL,CANCEL) the transaction. If user responds with either one of those request, I have to process the transaction according to request. But if he become unresponsive, the job processor will read the transaction from Queue (has timed delay) and failback the transaction. The main problem is the race condition between Processor and user response. Suppose if user responds around 120 secs later the reader will get the transaction and processes it, bur meanwhile, worker node also can get the work and process it same time — Nasantogtokh Amarsaikhan, Jul 19 '21 at 02:06

score 1 · Answer 1 · edited Oct 28 '21 at 13:17

I guess you should change your architecture a little. Worker Node does exactly what it is intended for, it takes jobs and runs them. So instead of blocking the queue in some way, you should only add the job to the queue when the user approved/canceled/failed it (or did not sent a response after 120 seconds).

If I understood you right, this should give you an idea how to have control over jobs between different requests:

// this is YOUR queueu object. I don't now implentation but think 
// of it like this..
const queue = new Queue()

// a variable holding the pending jobs which are not timeouted
// or explicitly approved/canceled/failed by user
const waitingJobs = {

}


// This could be your location where the user calls the api for creating a job.
app.post('/job', (req, res) => {

    // create the job as the user requested it
    const job = createJob(req)

    // Add a timeout for 120 seconds into your waitingJobs array. 
    // So if the user does not respond after that time, the job will 
    // be added to queue! .
    const timeout = setTimeout(() => {

        queue.add(job)

        // remove the reference after adding, garbage collection..
        waitingJobs[job.id] = null

    // job is added to queue automatically after 120 seconds
    }, 120 * 1000)

    // store the timeout in the job object!
    job.timeout = timeout

    // store the waiting job!
    waitingJobs[job.id] = job

    // respond to user, send back id so client can do another 
    // request if wanted.
    req.status(200).json({ message: 'Job created!', id: job.id })
})


app.post('/job/:id', (req, res) => {

    const id = req.params.id

    if (!id) {
        req.status(400).json('bad job id provided')
        return
    }

    // get the queued job:
    const job = waitingJobs[id]

    if (!job) {
        req.status(400).json('Job nod found OR job already processed. Job id: ' + id)
        return
    }

    // now the user responded to a specific job, clean the 
    // timeout first, so it won't be added to queue!
    if (job.timeout) {
        clearTimeout(job.timeout)
    }

    // Now the job won't be processed somewhere else!
    // you can do whatever you want...
    // example:

    // get the action
    const action = req.query.action
    
    if(!action) {
        res.status(400).json('Bad action provided: ' + action)
        return
    }

    if(action === 'APPROVE') {
        // job approved! , add it to queue so worker node 
        // can process it..
        queue.add(job)
    }

    if(action === 'CANCEL') {
        // do something else...
    }
    /// etc..

    
    // ofc clear the job reference after you did something..
    waitingJobs[job.id] = null
    

    // since everything worked, inform user the job will now be processed!
    res.status(200).json('Job ' + job.id + 'Will now be processed')
})

bull queue block job while got job on different function

1 Answers1