Stack Overflow
Stack Overflow

Questions tagged [bullmq]

BullMQ - Premium Message Queue for NodeJS based on Redis

BullMQ is a Node.js library that implements a fast and robust queue system based on Redis. The library is designed so that it will fulfill the following goals:

  1. Exactly once queue semantics, i.e., attempts to deliver every message exactly one time, but it will deliver at least once in the worst-case scenario*.
  2. Easy to scale horizontally. Add more workers for processing jobs in parallel.
  3. Consistent.
  4. High performant. Try to get the highest possible throughput from Redis by combining efficient .lua scripts and pipelining.
77 questions
0
votes
0 answers

Redis database gets randomly wiped

I'm having a strange Redis issue that causes my redis db0 to get completely wiped periodically for no obvious reason. I'm using redis with Bull queue for NestJS and inserting volatile keys with 7 days duration. Everything is hosted on Google Cloud…
Brlja
  • 364
  • 3
  • 14
0
votes
1 answer

Downloading large files from a Google Cloud bucket in a BullMQ worker leads to stalls

I've created a job scheduler/runner using BullMQ. I have a worker that, when handling a job, downloads a given file from a Google Cloud Bucket and then analyzes it. However, whenever the file to download is very large (>10GB), it can take several…
Rayhan Memon
  • 146
  • 4
  • 18
0
votes
1 answer

Bull - Triggering multiple functions on a single queue

i would like to create one Bull Queue, and put function in queue? is it possible. for example: I have http request /func1 + params -> start func1(params) http request /func2 + params -> start func2(params) http request /func3 + params -> start…
user1862965
  • 327
  • 3
  • 15
0
votes
1 answer

How does a worker know which queue to get tasks from?

Am working on a nodejs project where i need to implement task queueing. I have picked the bullMq with redis packages for this. Following the documentation here import { Queue, Worker } from 'bullmq' // Create a new connection in every…
drogon98
  • 13
  • 1
  • 5
0
votes
1 answer

how to serialize jobs in BullMQ

How to serialize jobs in job queue of BullMQ? I need that the next job is scheduled for execution only when the previous job is finished. I am using BullMQ 3.
Vitaly
  • 1
0
votes
1 answer

Bull generate 2000 files Memory leak node.js

I currently have a problem generating two thousand pdf's using node.js and bull. Indeed, I manage to generate about 400 pdf. Then I got this error message: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap…
SLSofiane
  • 49
  • 5
0
votes
0 answers

NestJS queues with more than one job

I have an issue with NestJS and Bull MQ, whenever I try to process more than one job only the very first one gets executed but not the rest. I go through emails then start processing them like this. only the first one is added to the queue, not the…
Mohamed Said
  • 524
  • 7
  • 27
0
votes
1 answer

Using BullMQ with NestJs, is it possible to use a config variable as part of a Job name?

I'm trying to use an environment variable value in the nestjs/bull module's @Process() decorator, as follows. How should I provide the 'STAGE' variable as part of the job name? import { Process, Processor } from '@nestjs/bull'; import { Inject }…
Aadmaa
  • 829
  • 5
  • 13
0
votes
1 answer

Is there a way to pass command line arguments to node process while loading environment to child process for a job in bull?

I need to pass command-line arguments or params or execargv which should come in process params while loading environment for a child process which will be launched by a bull for process a job. is it possible? if yes is there any way to do it? I can…
Stephen
  • 834
  • 7
  • 13
0
votes
0 answers

Github CI blowing up memory

I have a CI setup in Github which runs all the tests I currently have (200 approx) which are e2e tests using jest and superagent, following NestJS's standard testing pattern, also meaning that I have one or more test suites (files) for each…
SeekanDestroy
  • 511
  • 1
  • 5
  • 12
0
votes
0 answers

Looking for a Node.js queue system that allows only one consumer at a time (similar to Single Active Consumer in RabbitMQ)

My situation requires that the next message will only start processing after the previous one finishes processing. (The message processing function is an async function). RabbitMQ fits my needs with Single Active Consumer functionality and by…
shapeare
  • 4,133
  • 7
  • 28
  • 39
0
votes
2 answers

Bull's processor in NestJS never handled queued jobs

I am trying to send DTOs to my Redis queue with a Bull framework and handle these DTOs in processors. Sometimes job pass to processor (1 of 100) but most of the time failed with error: job stalled more than allowable limit and I have no idea how to…
Denis Stephanov
  • 4,563
  • 24
  • 78
  • 174
0
votes
1 answer

Does MikroORM require a RequestContext when used with background workers like BullMQ?

I'm using MikroORM with BullMQ workers. When MikroORM is used with Express apps, it requires a RequestContext in order to maintain unique identity maps for each request. I suspect the same is required when processing multiple jobs with the same…
calvinyoung
  • 346
  • 2
  • 12
0
votes
1 answer

NodeJS Express API - Ticketing/Queue System

Rephrased at the end NodeJS communicates with other APIs through GRPC. Each external API has its own dedicated GRPC connection with Node and every dedicated GRPC connection has an upper bound of concurrent clients that it can serve simultaneously…
GeorgePal
  • 85
  • 6
0
votes
1 answer

BullMQ start Job from file with IIFE

Let's say I have a function1.js (or ts, it doesn't matter in this case),function2.js and any other files which are IIFE with different logic, like that: (async function F() { try { //[1,2,..n].map(x => console.log(x)); //await any other…
AlexZeDim
  • 3,520
  • 2
  • 28
  • 64
1 2 3 4
5
6