Using Cheerio and Response for Node web scraper, passing response function result to the view

Question

I'm using this tutorial : https://www.smashingmagazine.com/2015/04/web-scraping-with-nodejs/

To make a really basic node web scraper. I have my app set up and running here:

What it is currently doing behind the scenes in my node app, is using the two modules cheerio and request to run this function (below). This function basically takes a URL, makes the request and grabs an element of the page with a data variable, scrapes the value of it, and logs the value, (temperature) to the console of my terminal of my computer. I need a way to send this value to my view and render it on the page instead of just logging it to the console of my computer.

The problem I'm having is that the scope of the request function below, I can't pass any of the return values, (temperature) to my view. I know there is something wrong with my set up because I currently have the request function INSIDE of my router.get. If I put the request function outside the router.get, I still cant pass the values to my view but it will successfully get the data from the web url and log it to my terminal's console. I hope I am being clear. Please see the res.render to my view which wraps the request function that is doing the web scraping..

router.get('/', function(req, res, next) {

    //target url we are scraping data from
    var url = "http://www.wunderground.com/cgi-bin/findweather/getForecast?&query=" + 02888;
    var temperature;
    // request function that uses the request module
    request(url, function (error, response, body) {

        if (!error) {
            // using cheerio module to load body of html req document and scrape the data
            var $ = cheerio.load(body),
                temperature = $("[data-variable='temperature'] .wx-value").html();
            // logs it to console (of my computer..)    
            console.log("It’s " + temperature + " degrees Fahrenheit.");
        } 

        else {
            console.log("We’ve encountered an error: " + error);
        }

        return temperature;
    });
    /* renders my view, this is my attempt to pass the values as variables to my handlebars template. Currently it is only passing the URL var as it exists before the request function call, and the empty var temperature (which gets its value during, and inside the request function call). How can i get at those values returned from the request function and pass to my view below? */
    res.render('index', {title: url, data: temperature } );  

});

score 1 · Accepted Answer · answered Jan 08 '16 at 20:53

The function in request is a executed asynchronously, so as you have it, render is getting called before temperature is set. You need to move the render function into the async function.

router.get('/', function(req, res, next) {

//target url we are scraping data from
var url = "http://www.wunderground.com/cgi-bin/findweather/getForecast?&query=" + 02888;
var temperature;
// request function that uses the request module
request(url, function (error, response, body) {

    if (!error) {
        // using cheerio module to load body of html req document and scrape the data
        var $ = cheerio.load(body),
            temperature = $("[data-variable='temperature'] .wx-value").html();
        // logs it to console (of my computer..)    
        console.log("It’s " + temperature + " degrees Fahrenheit.");
    } 

    else {
        console.log("We’ve encountered an error: " + error);
    }

    res.render('index', {title: url, data: temperature } );  
});

});

Using Cheerio and Response for Node web scraper, passing response function result to the view

1 Answers1