Nodejs - Need to perform around 100 API Async request

Question

I am new for nodejs and I trying to perform around 100 API request by using axios npm in single request. What will be best code logic to capture all response in better way of performance / error tracking.

Need to capture all response in single JSON file, so I decided to use createWriteStream('filename.json') to avoid the memory issue.

I tried something

const axios = require('axios');
const fs = require('fs');
const config = require('./config/secret.json');

app.get('/json', (req,res) => {

    const linkArr = ['https://apirequest1.com','https://apirequest2.com','https://apirequest3.com','https://apirequest4.com', '...'];
    const wipArr = [];

    for(let getAPI of linkArr){
        axios({
            method: 'get',
            url: getAPI,
            auth: {username: config.username, password: config.secret}
        })
        .then(function (response){
            const writeStream = fs.createWriteStream('wip.json');
            writeStream.write(JSON.stringify(response.data));
        })
        .catch(function (error){
            console.log(error);
        })
    }
    res.send('successfully saved all response');
});

Capture all API response in single hit and save them in array after completing the API request, need to write all response in JSON file.

Thanks in advance!

does `response.data` show something? i think axios needs `data` only for post — bill.gates, Nov 20 '20 at 07:25
@Ifaruki yes I getting the API response, we have the data keyword in response. — Anand Rajagopal, Nov 20 '20 at 07:34

kg99 · Accepted Answer · 2020-11-20T11:20:59.450

The first issue you have is that you create the stream everytime. This will overwrite the contents each time the promise is resolved. Remove this line.

const writeStream = fs.createWriteStream('wip.json');

You will have something like this.

const axios = require('axios');
const fs = require('fs');
const config = require('./config/secret.json');
const writeStream = fs.createWriteStream('wip.json');

app.get('/json', (req,res) => {


    const linkArr = ['https://apirequest1.com','https://apirequest2.com','https://apirequest3.com','https://apirequest4.com', '...'];
    const wipArr = [];

    for(let getAPI of linkArr){
        axios({
            method: 'get',
            url: getAPI,
            auth: {username: config.username, password: config.secret}
        })
        .then(function (response){
            //const writeStream = fs.createWriteStream('wip.json'); // remove this line because it will overwrite the file for each response.
            writeStream.write(JSON.stringify(response.data));
        })
        .catch(function (error){
            console.log(error);
        })
    }
    res.send('successfully saved all response');
})

;

EDIT: To wait for all requests, You can try something like this.

app.get('/json', async (req, res) => {
    let resp = null;
    const writeStream = fs.createWriteStream('wip.json');
    const linkArr = ['https://apirequest1.com', 'https://apirequest2.com', 'https://apirequest3.com', 'https://apirequest4.com', '...'];
    const promises = [];
    for (let getAPI of linkArr) {
        promises.push(makeCall(getAPI));
        resp = await Promise.all(promises); // resp is array of responses

       // for (let i = 0; i < resp.length; i++) {
          //  writeStream.write(JSON.stringify(resp[i], null, 4)); // to //format the json string
      //  }
    }
 for (let i = 0; i < resp.length; i++) {
                writeStream.write(JSON.stringify(resp[i], null, 4)); // to format the json string
            }
    res.send('successfully saved all response');
});


function makeCall(getAPI) {
    axios({
            method: 'get',
            url: getAPI,
            auth: { username: config.username, password: config.secret }
        })
        .then(function(response) {
            return response.data;
        });
}

I have not tested it but something along those lines. This will run all the requests.

To format JSON strings you can use.

JSON.stringify(resp[i], null, 4).

Have a look at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify

Edit: The problem was that the writeStream.write(JSON.stringify(resp[i], null, 4)); was inside the loop. Moved it outside.

Added code without testing. This should work for you.

app.get('/json', async(req, res) => {
    const writeStream = fs.createWriteStream('wip.json');
    const linkArr = ['https://apirequest1.com', 'https://apirequest2.com', 'https://apirequest3.com', 'https://apirequest4.com', '...'];
    const promises = [];
    for (let getAPI of linkArr) {
        promises.push(makeCall(getAPI));
    }
    const resp = await Promise.all(promises); // resp is array of responses
    for (let i = 0; i < resp.length; i++) {
        writeStream.write(JSON.stringify(resp[i], null, 4)); // to format the json string
    }
    res.send('successfully saved all response');
});



 function makeCall(getAPI) {
    return axios({
        method: 'get',
        url: getAPI,
        auth: { username: config.username, password: config.secret }
    })
    }

thanks for the response, I can able to save the response in JSON file. After saving every request in JSON file. Facing JSON formatter issue also I need to send response to client after completing the 100 request process. — Anand Rajagopal, Nov 20 '20 at 07:45
getting below error (node:14396) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_ARG_TYPE]: The "chunk" argument must be of type string or an instance of Buffer or Uint8Array. Received undefined at WriteStream.Writable.write (_stream_writable.js:285:13) at C:\Users\ranand20\myapp\app.js:105:29 at processTicksAndRejections (internal/process/task_queues.js:93:5) — Anand Rajagopal, Nov 20 '20 at 09:10
writeStreamFile.write(JSON.stringify(resp[i], null, 4)); this line throws error — Anand Rajagopal, Nov 20 '20 at 09:11
There is a problem with your response. Its probably not a valid JSON. Can you post the output before it fails. — kg99, Nov 20 '20 at 09:25
got it, actually writeStreamFile.write(JSON.stringify(resp[i], null, 4)); this line executing before getting the response. — Anand Rajagopal, Nov 20 '20 at 09:56
Great, but i doubt that was the problem. const resp = await Promise.all(promises); will wait for all responses to complete. Just remeber that with this method, if one fails, you have a problem. — kg99, Nov 20 '20 at 10:06
As of now I am trying with 3 API request only. I tried with postman its returning the response. — Anand Rajagopal, Nov 20 '20 at 10:11
I dont understand. Is it working or are you still having problems? — kg99, Nov 20 '20 at 10:36
Its not working actually ```writeStream.write(JSON.stringify(resp[i], null, 4))``` line is being executed before getting the response. ```const resp = await Promise.all(promises)``` is not waiting with promises array. — Anand Rajagopal, Nov 20 '20 at 10:44
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/224841/discussion-between-anand-and-kg99). — Anand Rajagopal, Nov 20 '20 at 10:49

Nodejs - Need to perform around 100 API Async request

1 Answers1