3

I'm using the async module to iterate over multiple urls with nightmarejs. I can't create a new nightmare instance because I will have to reauthenticate each time.

So I'm trying to use the async module. I'm getting a (I think, classic) problem the url for all iterations being the final url in the array - not each individual url. I thought using the async module would fix this (I also tried using let) but I'm still getting the problem

'use strict'

var Nightmare = require("nightmare");
var async = require("async");

//Creates the authenticated nightmare instance

var scraper = new Nightmare()
  .goto('https://www.example.com/signin')
  .type('#login', 'username')
  .type('#password', 'password')
  .click('#btn')
  .run(function(err, nightmare) {
    if (err) {
      console.log(err);
    }
    console.log('Done.');
  });

//Trying to use async module to iterate through urls

function load(url, callback){
  scraper
  .goto(url)
  .wait(2000)
  .screenshot('pic'+url[25]+'.png')
  .run(function(err, nightmare) {
    if (err) {
      console.log(err);
    }
    console.log('Done with ', url[25]);
    callback()
  }); 
}

var urls = [
  'https://www.example.com/p1',
  'https://www.example.com/p2',
  'https://www.example.com/p3',
]

async.each(urls, load, function (err) {
  console.log('done!');
});

Thanks for any advice

michaelbamberger
  • 97
  • 1
  • 5

1 Answers1

4

The problem lay with this line:

async.each(urls, load, function (err) {

By default, async runs each in parallel (see the documentation here, if you're curious). Nightmare is not capable of executing multiple requests in parallel, and as such, doing so will result in errant results like what you're seeing.

The solution is straightforward: switch to using async.eachSeries. That will guarantee your requests will be run in series, allowing Nightmare to work as designed.

Ross
  • 2,448
  • 1
  • 21
  • 24