How to execute commands in backend(Node.js Express) synchronously step by step?

Question

I want to do this. But I am not able to run all of this synchronously. Every step is executing all at once. Can anyone help?

I need to do this backend task in Node.js express.

Step 1: Enter the keyword from the front end(written in react)

Step 2: Run the command: " shell.exec(__dirname + /run.sh "${data}"); " at the backend. [[data is the keyword received from the front end]]. It will search the keyword in my file and generate a csv file.

Step 3: Run a python script which will execute a python file(csv_to_html.py) after completion of Step 2 execution. It will convert the csv file generated to html table and create a output.html file

Step 4: On completion of step 3, an output.html file will be generated. Send this file in an iframe on the front screen(index.html or index.ejs or index.hbs) below the search box. Suggestions are welcome on how can I display it on the frontend.

All this should be done dynamically.

This is what I have done:

    const { response } = require("express");
    const express = require("express");
    const bodyParser = require("body-parser");
    const shell = require("shelljs");

    const app = express();
    const spawn = require("child_process").spawn;

    app.use(express.static("public"));
    app.use(bodyParser.urlencoded({ extended: true }));
    var data = "";

    app.get("/", function (req, res) {
      shell.exec(__dirname + `/run.sh "${data}"`);
      const pythonProcess = spawn('python', ["csv_to_html.py"]);
      res.sendFile(__dirname + "/index.html");
    });

    app.post("/", function (req, res) {
      data = req.body.load_data;
      res.redirect("/"));
    });

    app.listen(3000, function () {
      console.log("Starting server at port 3000...");
    });

Why do you want to use the python program to convert the CSV into HTML instead of just doing that in Node? Also, could you add the work you've done so far? Do you have any part of this backend already developed or at least scaffolded? — lucasreta, Feb 14 '21 at 00:02
I am using python to convert it because I am new to Nodejs and don't have any idea how to do it. This is what I could think of! @lucasreta — Sudip Bala, Feb 14 '21 at 00:28
could you add an explanation for what the run.sh command and the python script do? — lucasreta, Feb 14 '21 at 00:30
run.sh will take the keyword and process it at my database and produce a csv file containing two columns in each row. There will be multiple rows. I need this csv file in a tabular format below the search box at front end. The python script(converts the csv file to html table and saves it to output.html) produces an output.html file which I tried to embed it in an iframe at front end. — Sudip Bala, Feb 14 '21 at 00:35
awesome, I think that's enough info to work on an answer, I'll get to it — lucasreta, Feb 14 '21 at 00:45

lucasreta · Accepted Answer · 2021-02-14T01:14:08.877

Enable json parsing of the body data.
Move execution of scripts to our POST request, so that they only run when new data is sent to the server.
Remove shelljs and rely only on Node.JS' own child_process to invoke commands.
Nest our callbacks so that they execute in the desired order.

Following these steps, you'd end up with the following file (comments with numbers added in the lines modified):

const express = require("express");
const bodyParser = require("body-parser");

const app = express();
// Modification #3
const { exec } = require("child_process");

app.use(express.static("public"));
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json()); // Modification #1

app.get("/", function (req, res) {
  res.sendFile(__dirname + "/index.html");
});

app.post("/", async function (req, res) {
  // Modification #2
  const data = req.body.load_data;
  // Modification #3 & #4
  exec(`./run.sh "${data}"`, async (err, stdout, stderr) => {
    if (err) console.error(err)
    console.log(stdout);
    exec('python csv_to_html.py', (err, stdout, stderr) => {
      console.log(stdout);
      res.redirect("/");
    });
  });
});

app.listen(3000, function () {
  console.log("Starting server at port 3000...");
});

The body of the request sent to the POST route should be of Content-Type application/json, and similar to the following:

{
    "load_data": "argumentForRunSH"
}

How to execute commands in backend(Node.js Express) synchronously step by step?

1 Answers1