I'm working on an express app that uses several Bull queues in production. We created a wrapper around BullQueue
(I added a stripped down version of it down below)
import logger from '~/libs/logger'
import BullQueue from 'bull'
import Redis from '~/libs/redis'
import {report} from '~/libs/sentry'
import {ValidationError, RetryError} from '~/libs/errors'
export default class Queue {
constructor(opts={}) {
if (!opts.label) {
throw new ValidationError('Cannot create queue without label')
}
if (!this.handler) {
throw new ValidationError(`Cannot create queue ${opts.label} without handler`)
}
this.label = opts.label
this.jobOpts = Object.assign({
attempts: 3,
backoff: {
type: 'exponential',
delay: 10000,
},
concurrency: 1,
// clean up jobs on completion - prevents redis slowly filling
removeOnComplete: true
}, opts.jobOpts)
const queueOpts = Object.assign({
createClient: function (type) {
switch (type) {
case 'client':
return client
case 'subscriber':
return subscriber
default:
return new Redis().client
}
}
}, opts.queueOpts)
this.queue = new BullQueue(this.label, queueOpts)
if (process.env.NODE_ENV === 'test') {
return
}
logger.info(`Queue:${this.label} created`)
}
add(data, opts={}) {
const jobOpts = Object.assign(this.jobOpts, opts)
return this.queue.add(data, jobOpts)
}
}
Then I created a queue that is supposed to send a GET
request using node-fetch
import Queue from '~/libs/queue'
import Sentry from '~/libs/sentry'
import fetch from 'node-fetch'
class IndexNowQueue extends Queue {
constructor(options = {}) {
super({
label: 'documents--index-now'
})
}
async handler(job) {
Sentry.addBreadcrumb({category: 'async'})
const {permalink} = job.data
const res = await fetch(`https://www.bing.com/indexnow?url=${permalink}&key=${process.env.INDEX_NOW_KEY}`)
if (res.status === 200) {
return
}
throw new Error(`Failed to submit url '${permalink}' to IndexNow. Status: ${res.status} ${await res.text()}`)
}
}
export default new IndexNowQueue()
And then this Queue is being added in the relevant endpoint
indexNowQueue.add({permalink: document.permalink})
In the logs I can see that the Queue is added, however, unlike the other queues (for instance aggregate-feeds
) it never moves forward
No error is thrown and any debugger breakpoint I added in there never gets reached. I also tried isolating the handler
function outside of the queue and it works as I would expect. What could be causing this issue? What other ways do I have to debug Bull?
It's worth mentioning that there are half a dozen queues in the projects and they are all working as expected.