1

I've created an NodeJS service that reads messages from an ActiveMQ queue, set up remotely, and picks up these messages, processes it and pushes some data to a GUI.

The issue i'm facing is while trying to process these messages on NodeJS end when they come in at a rapid pace to the ActiveMQ queue at approx. 5 messages (JSON) per second and each JSON message is around 18kb in size. The messages that come in are written to a file, saved to an intermediate MSSQL table and once saved, the JSON file is moved to a Processed folder.

The environment setup is:

  • NodeJS version 8.9.4.
  • ActiveMQ version 5.15.4.
  • Java version 1.8.0_171.
  • Three worker processes are created along side the main Node process to handle the message loads.

The workers are created using the node module "Workerpool". https://github.com/josdejong/workerpool

RAM: 2 GB
Processor: Intel Xeon Dual Core processor @2.27GHz 
OS: Windows Server 2008 R2

I keep running into a Unhandled rejection Range Error: Out of Memory Exception, after processing around 3000 messages from ActiveMQ queue, messages pushed to the queue from an external source at frequency of 200ms / message.

The code:


message.readString('utf-8', function (err, body) {
  fs.writeFile('/path/to/writefileto', JSON.stringify(body), function(err) {
  if(err) {
    // handle the error
  } else {
    /* if the file exists in the source path, move to a Processed path */
    if(fs.existsSync('/path/writtento')) {
      fs.rename('/path/writtento', '/path/to/processedDir', function(err) {
         if(!err) {
           console.log("Successful in moving the file to Processed path");
         }
      }
    }
  }

What could be causing this issue?

Please let me know if any other information is required.

sanmyth
  • 85
  • 5

0 Answers0