shelljs execution gets stuck on DigitalOcean

Question

I have a React web app hosted on DigitalOcean (Ubuntu 16.04). The app has an R script on the backend (node) that is invoked by shelljs (shCMD command invokes the R script. It never gets inside the exec callback):

exec_full_analysis = shell.exec(shCMD, function(code, stdout, stderr) {
 if (code === 0) {
      console.log('Program output:', stdout);

      pdfService.drawChartsPDF(typeArr, analysisId, datasetArr, pairArr, filterArr);

      console.log('sendStatus 200')

      res.sendStatus(200);
    } else {
      console.log('Program stderr:', stderr);
      //res.sendStatus(500);
    }
  });

I am using foreverjs to start the server. It is running in a conda environment. So, when the script starts running initially I see correct log output (R script is running) that foreverjs provides me with but very quickly log output stops being produced. No errors neither in the browser, nor in the forever log output are present. The server gets to a certain stage in my R script and somehow gets stuck. Is there a way to figure out at all what is going wrong? What would you recommend? Locally everything is working just fine. The only thing that may be different is conda environment, but I would expect it giving an error but there is none.

Update

I looked into /var/log/syslog but I see no OOM errors: https://www.digitalocean.com/community/questions/python-script-gets-killed

Update

It is not an issue with foreverjs: running simple node server.js stucks at the same place in R script.

The script gets stuck in Seurat- package function ScaleData:

 seurat_object <- ScaleData(object = seurat_object, vars.to.regress = c("nUMI"))

with the following output:

ScaleData is running on non-normalized values. Recommended workflow is to run NormalizeData first. Regressing out: nUMI | | 0%

But, of course, locally it is running well. And on the server it gets stuck on the second for-loop iteration, so one time it runs just fine on the server, but the second time it fails as if there is a restricted time for running a process set on DigitalOcean.

Update

The server was apache2. I switched to nginx and now the server runs fine, it is not getting stuck and the issue is almost solved except that now I am getting 504 gateway timeout errors.

score 1 · Answer 1 · answered Mar 29 '19 at 05:06

So, I do not know why but it was apache2 server problem. I switched to nginx, but then RAM out of memory errors appeared which I fixed by reducing the amount of memory my R script uses:

options(java.parameters = "-Xmx6000m")

It was -Xmx8000m. Then the script was finishing correctly on the server, but on the client I saw 504 gateway timeout errors which I fixed by the solution I found here: https://asdqwe.net/blog/solutions-504-gateway-timeout-nginx/

Just created file at /etc/nginx/conf.d/timeout.conf with:

proxy_connect_timeout       3000;
proxy_send_timeout          3000;
proxy_read_timeout          3000;
send_timeout                3000;

After which it finally started working

shelljs execution gets stuck on DigitalOcean

1 Answers1