1

Well I (naively) tried to get bull working in a sails application: ultimatelly I wish to have a queue to which I can add/remove/check tasks based on incoming routes.

Now as I understand sails to create a queueing system that works globaly I would have to add this setup in bootstrap.js.

/**
 * Bootstrap
 * (sails.config.bootstrap)
 *
 * An asynchronous bootstrap function that runs before your Sails app gets lifted.
 * This gives you an opportunity to set up your data model, run jobs, or perform some special logic.
 *
 * For more information on bootstrapping your app, check out:
 * https://sailsjs.com/config/bootstrap
 */
module.exports.bootstrap = function(done) {

  // It's very important to trigger this callback method when you are finished
  // with the bootstrap!  (otherwise your server will never lift, since it's waiting on the bootstrap)
  let Queue = require('bull');
  let q = new Queue('test queue');
  q.process(function(job, done){
    console.log("starting job");
    for(let i = 0; i<job.value; i+=1) {
      console.log(i);
    }
    done();
  });
  q.add({'value':10});
  global.DirectUpdateQueue = q;
  return done();

};

Given above code, sails launches perfectly fine, and in the routes I can see the global.DirectUpdateQueue existing.

What does however not work is that the queued tasks are executed. - I do not see any log in the console ("starting job" is expected at least). Nor does the code break whe nI put a breakpoint in the processing function.

So what is going on here?

EDIT: can this be due to me not having set up a (local) redis server? - I don't find any information on this subject but I expected/hoped bull.js to actually handle this server internally and (even more importantly) not be limited to a specific (OS) environment.

paul23
  • 8,799
  • 12
  • 66
  • 149
  • have you had success using bull.js outside of sails.js? I'm not familiar with bull.js at all, but their documentation seems to indicate that it's dependent on redis ("Requirements: Bull requires a Redis version greater than or equal to 2.8.18.") – davepreston Jan 15 '18 at 19:36
  • @davepreston Using what server software then? I'd need something like express/sails incoming messages etc right? I can't just "run node" and connect to it? – paul23 Jan 15 '18 at 23:05

3 Answers3

1

So, first of all, you have to make sure you have Redis installed on your server. When creating a queue, you can pass Redis config in my example below it's the default.

Then in bootsrap.js:

  var Queue = require('bull');
  var testQueue = new Queue('Website Queue', 'redis://127.0.0.1:6379');
  testQueue.process(function(job, done){

    console.log('job started');

    setTimeout(function () {

        console.log('10 seconds later');
        console.log(job.data);

    }, 10000)

    done(); 
    });    

  global.testQueue = testQueue;

then from action/controller you can do this:

testQueue.add({'value':10});
Sam Axe
  • 437
  • 1
  • 8
  • 25
0

First you must connect to a Redis server

var testQueue = new Queue('test', {
  redis: {
    port: 6379,
    host: '127.0.0.1',
    password: 'secret'
  }
});

According to the doc :

If the queue is empty the job will be executed directly, otherwise it will be placed in the queue and executed as soon as possible.

To access data in job, use job.data object :

testQueue.process((job) => {
  console.log("job with data 'foo' :", job.data.foo);

  // example with Promise
  return asynchTreatment()
  .then(() => { console.log('treatment ok'); })
  .catch((err) => { console.log('treatment ko :', err); }
}).on('completed', (job, result) => {
  // Job completed with output result!
  console.log('result :', result);
});

testQueue.add({ foo : 'bar' });

EDIT 1 :

The doc says :

It creates a new Queue that is persisted in Redis. Everytime the same queue is instantiated it tries to process all the old jobs that may exist from a previous unfinished session.

So if the server restart, you don't lose your jobs.

Martial
  • 1,446
  • 15
  • 27
  • A Redis server is required to use bull. Where do you think bull is storing queue data ? Furthermore, it is quite simple to install one on your own. – Martial Jan 16 '18 at 21:38
  • Well I'd hoped it would just keep the data in memory. Mostly since if the server restarts/has a hickup it's required to empty the queue list anyways - the queue is also supposed to be "almost always empty" except for very short times when suddenly many things are added (typically 9 o'clock in the morning). So the setup of an extra redis server feels overkill. – paul23 Jan 17 '18 at 09:18
  • Why an "extra" server ? You need at least one. And the queue is not empty on server restart. I edited my answer for that – Martial Jan 17 '18 at 17:13
0

Just use job.data.value in your for loop

for(let i = 0; i<job.data.value; i+=1) {
  console.log(i);
}
Bhaumik Gandhi
  • 223
  • 2
  • 6