Graphicsmagick is taking more processing time on server

Question

I am using GraphicsMagick to colorize multiple layers of images and composing it to form single image and returning it.

On local it was taking around 30ms to process single layer (1000 x 1000).

Following is the image.

Following is my colorization code:

var colorizeImage = function(imageName, hex, callback) {
  //Setting image path.
  var sourceImgSrc = getSourceImgSrc(imageName); //Source image path
  var convertedImgSrc = getConvertedImgSrc(imageName); // New image path
  var hslObj = getConvertedHSLFromHex(hex);
  var hue = hslObj.h;
  var saturation = hslObj.s; 
  var lightness = hslObj.l; 
  try{
      //Query  
        var modulateImg = "gm convert -resize 500 "+sourceImgSrc+" -modulate "+lightness+","+saturation+","+hue+" -limit memory 100KB "+convertedImgSrc; 

        exec(modulateImg, function(err) {
          if (err) {
            callback('', err);
          } 
          else{
            callback(convertedImgSrc);
          }

        });
  }
  catch(err){
    callback('', err);
  }
}

But, When I have hosted this to my server and performed load testing with only 50 concurrent users load. Performance for single layer gets degraded to around 250ms for single layer.

Following is my Server Config:

GraphicsMagick: 1.3.26

OS: Window server 2012

Processor: Intel(R) Xeon(R) 2.40 GHz (2 Processors)

RAM: 128GB

Please help me to improve performance so that it can handle huge load easily.

Thanks.