0

I'm working on an express app that uses several Bull queues in production. We created a wrapper around BullQueue (I added a stripped down version of it down below)

import logger from '~/libs/logger'
import BullQueue from 'bull'
import Redis from '~/libs/redis'
import {report} from '~/libs/sentry'
import {ValidationError, RetryError} from '~/libs/errors'

export default class Queue {

  constructor(opts={}) {
    if (!opts.label) {
      throw new ValidationError('Cannot create queue without label')
    }

    if (!this.handler) {
      throw new ValidationError(`Cannot create queue ${opts.label} without handler`)
    }

    this.label = opts.label

    this.jobOpts = Object.assign({
      attempts: 3,
      backoff: {
        type: 'exponential',
        delay: 10000,
      },
      concurrency: 1,
      // clean up jobs on completion - prevents redis slowly filling
      removeOnComplete: true
    }, opts.jobOpts)

    const queueOpts = Object.assign({
      createClient: function (type) {
        switch (type) {
          case 'client':
            return client
          case 'subscriber':
            return subscriber
          default:
            return new Redis().client
        }
      }
    }, opts.queueOpts)

    this.queue = new BullQueue(this.label, queueOpts)

    if (process.env.NODE_ENV === 'test') {
      return
    }

    logger.info(`Queue:${this.label} created`)
  }

  add(data, opts={}) {
    const jobOpts = Object.assign(this.jobOpts, opts)
    return this.queue.add(data, jobOpts)
  }
}

Then I created a queue that is supposed to send a GET request using node-fetch

import Queue from '~/libs/queue'
import Sentry from '~/libs/sentry'
import fetch from 'node-fetch'

class IndexNowQueue extends Queue {

  constructor(options = {}) {
    super({
      label: 'documents--index-now'
    })
  }

  async handler(job) {
    Sentry.addBreadcrumb({category: 'async'})

    const {permalink} = job.data

    const res = await fetch(`https://www.bing.com/indexnow?url=${permalink}&key=${process.env.INDEX_NOW_KEY}`)

    if (res.status === 200) {
      return
    }

    throw new Error(`Failed to submit url '${permalink}' to IndexNow. Status: ${res.status} ${await res.text()}`)
  }
}

export default new IndexNowQueue()

And then this Queue is being added in the relevant endpoint

indexNowQueue.add({permalink: document.permalink})

In the logs I can see that the Queue is added, however, unlike the other queues (for instance aggregate-feeds) it never moves forward

enter image description here

No error is thrown and any debugger breakpoint I added in there never gets reached. I also tried isolating the handler function outside of the queue and it works as I would expect. What could be causing this issue? What other ways do I have to debug Bull?

It's worth mentioning that there are half a dozen queues in the projects and they are all working as expected.

ilrock
  • 573
  • 8
  • 24

0 Answers0