How to use request within async.each in node.js Express

Question

In my node.js project, I need to make iterative API request so I'm using async.each method.

The console complains that url is not defined and I understand this because I've only declared urls which is an array of url.

I'm just not sure how I can put together request() from request module inside async.each. To satisfy async.each() I've placed urls as the first argument but request() itself requires a query string argument which is url.

I'm also using _.extend from Underscore to merge two responses but I'm not sure where I have it currently is in the right place.

Can someone please point me in the right direction?

var urls = [];

Object.keys(result.items).forEach(function(item) {
  urls.push("https://www.googleapis.com/youtube/v3/videos?part=contentDetails&id=" + result.items[item].contentDetails
    .videoId + "&key=xxx");

})

async.each(
  urls,
  request(url, function(err, response, body) {
    if (err) {
      console.log(err);
      return;
    }
    body = JSON.parse(body);
  }),
  function() {
    var extended = _.extend(result, body);
    getVideoDetailsCallback(null, extended)
  });

score 1 · Accepted Answer · answered Dec 22 '15 at 03:22

1

It seems you're calling request with callbacks and all, and not just referencing it, which means you probably need an anonymous function call.

Also, if you want an array at the end, or whatever you're extending, you could just use async.map instead, and do something like

var urls = Object.keys(result.items).map(function(item) {
    return "https://www.googleapis.com/youtube/v3/videos?part=contentDetails&id=" + result.items[item].contentDetails.videoId + "&key=xxx";
});

async.map(urls, function(url, callback) {
    request(url, function(err, response, body) {
        if (err) {
            return callback(err);
        }
        callback(null, JSON.parse(body));
    });
}, function(err, extended) {
    if (err) {
        // handle error
    }
    // extended is an array containing the parsed JSON
    getVideoDetailsCallback(null, extended); 
});

answered Dec 22 '15 at 03:22

adeneo

312,895
29
395
388

Thank you. It looks promising. I used _.extend because I need to merge `extended` containing the parsed JSON with the response from the first API request. This is my previous question that shows the structure using async.waterfall. http://stackoverflow.com/questions/34406490/node-js-dependant-api-request-calls-using-async-waterfall-and-merge-json-respo How do I merge JSON from this iteration with another JSON from other response? – Seong Lee Dec 22 '15 at 03:28
When using `.map` the data received in the final callback, `extented`, will be an array containing the data from all the requests made. I would probably modify the `getVideoDetailsCallback` function to take an array like that, and just reference that function as the final callback, but if you want to merge the final array you'll probably need something more like this -> http://stackoverflow.com/questions/27556728/how-can-i-merge-array-of-objects-with-underscore-js – adeneo Dec 22 '15 at 03:37

score 1 · Answer 2 · answered Dec 22 '15 at 03:26

If you want to get the results from each iteration. I suggest using async.map. Map returns an array of results to the final callback.

Here's an example,

var async = require('async');

async.map([ 1, 2, 3, 4 ], function ( num, cb ) {
  cb(null, num);
}, function (error, results) {
  console.log(results);
});
// output: [ 1, 2, 3, 4 ]

As for your code snippet, your second argument is the wrong type. You're invoking request which doesn't return a function. The second argument is expected to be a function so what you need to do is wrap the request around function definition.

async.map(urls, function(url, callback) {
  request(options, callback);
}, function (error, results) {
  // do something with results
});

How to use request within async.each in node.js Express

2 Answers2

Linked