Scrape data from website from on change event using node.js

Question

I'm trying scrape data from one website. In that when page load there is drop down list and I have to select specific value from the dropdown.

For scrapping data from web I'm using cheerio reference link is https://www.npmjs.com/package/cheerio.

When dropdown value change then they call some Javascript function same like this way https://www.w3schools.com/jsref/tryit.asp?filename=tryjsref_onchange.

I am able to change the dropdown value in node.js but value's are not change. When I scrape after change the value of the dropdown, I'm still getting default page load data.

I have search a lot on that but I didn't get any proper output. I need some module or code which is works fine with 'node.js'.

Cheerio just parses html. You need something with a javascript engine like Puppeteer or NightmareJS. — pguardiario, Mar 20 '18 at 23:59
I use `NightmareJS` and it works for me. Thank you @pguardiario — Sachin Shah, Mar 25 '18 at 05:27

score 0 · Accepted Answer · answered Mar 25 '18 at 05:36

Just recently I used nightmare.js and it's good to use it. For that you have to install it.

npm install --save nightmare

The main thing is that which I feel , You are able to debug the code from GUI. As per your code it will show in GUI form. It's feel like automation testing.

const Nightmare = require('nightmare')

const nightmare = Nightmare({ show: true })

If you want to close the GUI the just set {show:false}.

var URL = "www.example.com";
nightmare
  .goto(URL)
  .click('#btn1')
  .end()
  .then(console.log)
  .catch(error => {
     console.error('Search failed:', error)
  })

Scrape data from website from on change event using node.js

1 Answers1