4

In other languages this task is easy, just use something like the following in Python:

while True:
    # Program logic that will run forever
    time.sleep(5) # Make the program wait for a few seconds

My program will also want to pause execution if there is nothing to process. I don't want it to overload the database with queries.

Background

I am trying to write an application that will act as a background queue processor. It will check the database to see if there are any items in the queue that need to be processed and then this program will write the data to files on the disk. The data will be added to the database intermittently by users of a different system that connects to the same database.

I don't think the forever npm module is a good fit since that module simply checks to see if a script is running and if its not it will restart it and report any standard error or output to a file.

My Thoughts in Code

while(true){
   db.dataqueue.find({processed: 0}).count((err, count) => {
      if(count == 0){
          //Sleep here or don't check the db again for a while
      }else{
          //Do the processing on the datas and make files. Another database find.count
          //call should not happen until the queue is processed.
      }
}

I'm not sure how to make it sleep since the callback from mongojs can not affect the while loop. I've looked at doing this with a promise and making the the parent function of this async which might work, but I'm not understanding how to implement it from this answer: How to sleep the thread in node.js without affecting other threads?

Also I have thought about replacing the while(true) with setInterval or setTimeout to make the program persistent. This article has been helpful in understanding the event loop.

chipadmin
  • 79
  • 9
  • 1
    Your thoughts are basically correct. You can use a `while(true)` loop and `await` a promise inside it, or you can use `setTimeout` inside the callback. It seems like you're on the right track, but you don't really have a specific question here, which in my opinion makes it a bit too broad for Stack Overflow. I think you should try to get one of those approaches working and post a new question if you get stuck. – Paul Apr 23 '18 at 18:46
  • Thank you for your comment! I will try the promise technique out and see how it goes. – chipadmin Apr 23 '18 at 18:49
  • If you use [bluebird](http://bluebirdjs.com/docs/api/promise.delay.html) you could do `await Promise.delay(5000);` in your loop, or without bluebird you can make your own promise that resolves after a timer, something like `await new Promise( resolve => setTimeout( resolve, 5000 ) );` – Paul Apr 23 '18 at 19:04
  • 1
    I showed an example of something similar to this here: https://stackoverflow.com/questions/50011032/optimal-solution-to-creating-a-run-loop-in-javascript/50011213#50011213 – Paul Apr 24 '18 at 21:54

1 Answers1

0

I've found the solution to this thanks to Paulpro telling me I was on the right track. First off I switched to using mongoist which support promises and then I'm using the new async and await syntax to get the desired results: 

const snooze = ms => new Promise(resolve => setTimeout(resolve, ms));
getToProcessCount();
async function getToProcessCount(){
    while(true){
         const datacnt = await db.queue.count({processed: 0});
         console.log(datacnt);
         if(datacnt == 0){
             console.log("Nothing to do, sleep for a while");
             await snooze(2000);
             console.log("I just woke up");
         }
         else{
             console.log("Do some processing here");
         }
     }
 }

The async function allows this type of function to pause its execution and wait for a promise to return which is very useful when dealing with database queries and write it in a synchronous format. The await keyword is what will pause the execution until the promise is fulfilled and it actually returns data so that is why I'm able to get the records count from the mongodb collection and output it in the next line.

My other requirement was to force the program pause if there is not any data to process so that it doesn't send a bunch of queries to the database constantly and that is what the snooze function does. It is important to note the await in front of the snooze function or else the snooze will be executed and immediately the next line will run.

This video from Fun Fun Function really helped me understand the async function and await syntax.

chipadmin
  • 79
  • 9