Using a closure in a do while loop for a get request

Question

I have 5 endpoints I need to grab information from. I can only grab 500 objects at a time. My thought process is that I need to loop through each endpoint and push the returned result to my object. I then nest the request inside of a do while loop to make sure that the loop continues to until the entire obj. For example, if I have 50,000 candidates the do while loop would go through 100 times and push each result to the obj, then at after that is finished push that to the parentObj. I am running into the problem that loop finishes before the get request is returned due to the async nature. Since order doesn't matter, how would I ensure that the request returns successfully before the loop increments? I was reading into David's Article and closures work for a for loop but not a do while. How would a closure work in this case with a do while? Correct me if my logic is incorrect.

var dataArr = ['/v1/candidates?per_page=500&page=1','/v1/eeoc?per_page=500&page=1','/v1/jobs?per_page=500&page=1','/v1/job_stages?per_page=500&page=1','/v1/applications/?per_page=500&page=1'];
var parentObj = {};

for ( var i = 0; i < dataArr.length; i++ ) {

    var options = {
      hostname: 'myURL',
      port: 443,
      path: dataArr[i],
      method: 'GET',
      headers: {
         'Authorization' : 'Basic MYKEY',
      }
    };

    var obj = {};
    do {
       https.get(options, (res) => {
          res.on('data', (d) => {
             var result = d.toString('utf8');
             // push returned result object to obj 
          });
       }).on('error', (e) => {
         console.error(e);
       });
     } while (obj.length % 500 === 0 && obj.length ==! 0); 
     // push obj to parentObj here
    } 
};
callback(null, parentObj);

` I was reading into David's Article and closures work for a for loop but not a do while. ` closures work everywhere, but they are not the problem here. — Jonas Wilms, Aug 03 '18 at 13:48

score 1 · Accepted Answer · answered Aug 03 '18 at 13:47

The problem is not a closure, the problem is that the callback of the request will happen after the loop finishes, which it never does. So you have to await the request before you continue looping:

 (async function() {
   for ( var i = 0; i < dataArr.length; i++ ) {

      var options = {
        hostname: 'myURL',
        port: 443,
        path: dataArr[i],
        method: 'GET',
        headers: {  'Authorization' : 'Basic MYKEY'   }
      };

      var obj = {};
      do {
        try {
         const request = await new Promise(resolve => https.get(options, resolve));
         const result = await new Promise((resolve, reject) => {
            request.on("data", d => resolve(d.toString("utf8")));
            request.on("error", reject);
         });
        } catch(e) { /* handle request errors */ }
       } while (obj.length % 500 === 0 && obj.length ==! 0); 
       // push obj to parentObj here
      } 
      callback(null, parentObj);
  }
})();

The https library might provide some helpers to work with promises, but as I don't know them I created them manually

Thank you. This helped a lot! – Jghorton14 Aug 03 '18 at 14:03 — Jghorton14, Aug 03 '18 at 14:03

Using a closure in a do while loop for a get request

1 Answers1