1

The Goal:

Given an array of URLs, I need to send a request and get back the statusCode for each one. Finally, I need to match that statusCode with the request URL in an array of objects which can then be returned to the page.

Like this:

checkList: [
    {
      name: "Google",
      url: "https://www.google.com"
    },
    {
      name: "Microsoft",
      url: "https://www.microsoft.com"
    },
    {
      name: "Github",
      url: "https://www.github.com"
    }
  ]

For each URL in the list, I need to simply make an http request and get back a status code (eg: 200) and match that with the request path. So the final output should look like this...

results: [
    {
      name: "Google",
      url: "https://www.google.com",
      status: 200
    },
    {
      name: "Microsoft",
      url: "https://www.microsoft.com",
      status: 200
    },
    {
      name: "Github",
      url: "https://www.github.com",
      status: 200
    }
  ]

I'm currently using BluebirdJS and Needle to handle the async calls to get the response codes.

The problem is that I can't see a way to match the requesting URL with the response. Since each result comes back at different times (as intended) and the Needle response doesn't include the request URL, I can't say something like... "https://google.com response is 200". I can only get a list of response codes.

I'm relatively new to advanced JS, so maybe there's a way to override callbacks/promises which allow me to pass the request URL through the entire process, but I don't know how.

This is a Gist of the code I'm currently using based on demos/snippets I've found elsewhere... https://gist.github.com/sgelliott/13840b6f2f9d2ab481fcded6dc4a9188

The code also included inline below...

var needle = require('needle');
var Promise = require("bluebird");
Promise.promisifyAll(needle);

var allPaths = ["https://google.com", "https://microsoft.com", "https://github.com"];
var current = Promise.resolve();

Promise.map(allPaths, function(path) {
    current = current.then(function() {
        console.log("path: " + path); // path shows up here correctly - it's available
        return needle.getAsync(path, options);
    });
    return current;
}).map(function(resp, body) {
    return {
        path: "path goes here",
        status: resp.statusCode
    };
}).then(function(results) {
    console.log("results: " + JSON.stringify(results)); // this outputs 'resulst' : [{"path":"path goes here","status":200},{"path":"path goes here","status":302},{"path":"path goes here","status":200}]
    return renderResults(results); //this simply sets 'statusResults' to 'results' for the next res.send to use - will refine once I solve the path matching issue
}).then(function() {
    console.log('All Needle requests are processed!');
    res.send(statusResults); //
}).catch(function(e) {
    console.log(e);
});

I apologize if this is a duplicate of something else. I wasn't able to find it after a lot of research. If the solution is to use something other than Needle, that's perfectly fine. I'd prefer to stick with Bluebird though. Thank you in advance!

sgelliott
  • 150
  • 7
  • Why not put the code in the question itself, rather than an external link that will go stale, and then the question is useless to future readers? – Jaromanda X Mar 03 '17 at 03:54
  • in line 20 ... where you say `I have no way to access the original request URL/path here for matching` ... wouldn't `results[n]` be the result for `allPaths[n]` – Jaromanda X Mar 03 '17 at 03:57
  • @JaromandaX I wasn't ever planning on destroying the Gist, but I've adjusted the post to include it inline. As for your question... that's a good question. I'd assumed that since async calls may return values at different times given network traffic that there'd be no guarantee that the array of results would be in the same order as the original set of request URLs. Do you know whether such a thing is guaranteed? It seems there's be a more appropriate way to do this. – sgelliott Mar 03 '17 at 04:14
  • Have you looked at async.js as a means of doing things in order to ensure that you get the results back as you want them? http://caolan.github.io/async/ – idream1nC0de Mar 03 '17 at 04:17
  • I know native `Promise.all` guarantees it - and reading the bluebird [Promise.map](http://bluebirdjs.com/docs/api/promise.map.html) documentation *carefully* - yes it is guaranteed - don't let the last sentence bother you ... execution order isn't guaranteed, but that's how it is with asynchronous code :p – Jaromanda X Mar 03 '17 at 04:17
  • @JacobHeater - never mix promises with asyncjs – Jaromanda X Mar 03 '17 at 04:17
  • @JaromandaX I didn't see the updated question with code samples. – idream1nC0de Mar 03 '17 at 04:20
  • if the results from map were not guaranteed to be the same order, it would be misleading to call it `map` :p – Jaromanda X Mar 03 '17 at 04:20
  • 2
    @JacobHeater - wasn't having a go at you - just putting it out there :p – Jaromanda X Mar 03 '17 at 04:20
  • 1
    @JacobHeater - Thanks for the suggestion. Yes, I did try that out too. Not seeing any way with that library either to pass through the request path/url throughout the async promise chain. No matter what async method I use, the only data I can access is the resp, body results from the needle.get call. – sgelliott Mar 03 '17 at 04:34
  • @JaromandaX - I can see your point about map. I'll try that out. Although it still seems there has to be a way to overload a callback/promise with additional information. Otherwise even if it works in my case, there are other cases that would still exist. Also, it's tough to verify that the statusCode is really matching the URL since they'll mostly be similar except on rare errors. – sgelliott Mar 03 '17 at 04:38

1 Answers1

0

I've solved this myself and I hope the solution is of use for others. Here's the updated code that's working using .bind

module.exports.gethealth = function (req, res) {
  var statusResults = [];
  var resultCount = 0;
  var allPaths = [];
  for (var item in config.checkList) {
    allPaths[item] = config.checkList[item].url;
  }

  var options = {connection: 'keep-alive'};

  Promise.map(allPaths, function (path) {
    console.log("path: " + path);

    return needle.getAsync(path, options)
      .bind(this, path)
      .then(function (resp, body) {
        statusResults[resultCount] = {path:path, status:resp.statusCode};
        resultCount++;
      });
  }).then(function () {
    res.send(statusResults);
  });

};

Let me know if you have any follow-up questions.

sgelliott
  • 150
  • 7