1

I am trying to pass the util module object to puppeteer page.evaluate with no success. I understand this question was asked in How to pass required module object to puppeteer page.evaluate but the solution provided does not work in my case. MWE:

const puppeteer = require("puppeteer");

const path = "http://books.toscrape.com/";

// scrape funs 
(async () =>{
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();
    await page.goto(path, {waitUntil: "networkidle2", timeout: 0});
    await page.waitFor(1000);
    await page.addScriptTag({path: './node_modules/util/util.js'});
    // selector with replaceable element
    const buttonText = await page.evaluate(() => {
        let selectorButton = "#default > div > div > div > div > section > div:nth-child(2) > ol > li:nth-child(%s) > article > div.product_price > form > button";
        let buttons = [];
        for(let i = 1; i < 21; i ++){
            let textOut = document.querySelector(util.format(selectorButton, i)).innerText;
            buttons.push(textOut);
        };
        return buttons;  
    });

// return
await browse.close();
console.log(buttonText);
})();

Shows error:

UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Evaluation failed: ReferenceError: util is not defined

Thank you

Edit 1

Adding const util = require("util"); in the initial lines and doesn't work as shown in How to pass required module object to puppeteer page.evaluate.

Edit 2

Even when I use browserify I can't seem to inject the utilmodule into the puppeteer page. Steps:

On project PATH, create main.js as follows: var util = require('util');

Then on PATH in terminal: browserify main.js -o bundle.js. A file bundle.js appears in the project PATH.

Then run the following:

const puppeteer = require("puppeteer");
const path = "http://books.toscrape.com/";

// scrape funs 
(async () =>{
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();
    await page.goto(path, {waitUntil: "networkidle2", timeout: 0});
    await page.waitFor(1000);
    await page.addScriptTag({path: "main.js"});
    await page.addScriptTag({path: "bundle.js"});
    // selector with replaceable element
    const buttonText = await page.evaluate(() => {
        let buttons = [];
        let selectorButton = "#default > div > div > div > div > section > div:nth-child(2) > ol > li:nth-child(%s) > article > div.product_price > form > button";
        for(let i = 1; i < 21; i ++){
            let textOut = document.querySelector(util.format(selectorButton, i)).innerText;
            buttons.push(textOut);
        };
        return buttons;  
    });

// return
await browse.close();
console.log(buttonText);
})();

Error:

UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Evaluation failed: TypeError: Cannot read property 'format' of undefined at :5:55

JdeMello
  • 1,708
  • 15
  • 23

1 Answers1

2

You need to include a browser compatible build of ./node_modules/util/util.js. You can use browserify to do that or use their online service Browserify Wizard - util to download the browserified version.

Code on https://try-puppeteer.appspot.com/

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto("http://books.toscrape.com/", {waitUntil: "networkidle2", timeout: 0});
await page.waitFor(1000);

//Copy of https://wzrd.in/standalone/util@latest
await page.addScriptTag({url: "https://cdn.rawgit.com/brahma-dev/099d0d6d43a5d013603bcd245ee7a862/raw/b0c6bb82905b5b868c287392000dc2487c41994d/util.js"});

// selector with replaceable element
const buttonText = await page.evaluate(() => {
    let buttons = [];
    let selectorButton = "#default > div > div > div > div > section > div:nth-child(2) > ol > li:nth-child(%s) > article > div.product_price > form > button";
    for(let i = 1; i < 21; i ++){
        let textOut = document.querySelector(util.format(selectorButton, i)).innerText;
        buttons.push(textOut);
    };
    return buttons;  
});

// return
await browser.close();
console.log(buttonText);
Brahma Dev
  • 1,955
  • 1
  • 12
  • 18
  • could you expand on this answer please? I've added the url at `await page.addScriptTag({url: 'https://wzrd.in/standalone/util@latest'}); ` but got `Error: Loading script from https://wzrd.in/standalone/util@latest failed`. Thank you! – JdeMello Mar 22 '18 at 14:00
  • I also created a file `main.js` which is simply `var util = require("util");` then did `browserify main.js -o bundle.js` in the project path and added the modules with the `page.addScriptTag()` command. – JdeMello Mar 22 '18 at 14:30
  • You need download `https://wzrd.in/standalone/util@latest` and save it as a file and then include that file. Also, browserifying `main.js` exports `main.js` not `util`. You have to browserify `./node_modules/util/util.js` and use that bundle. – Brahma Dev Mar 22 '18 at 14:40
  • I've done both with no success. As I said, first attempt: `page.addScriptTag({url: 'https://wzrd.in/standalone/util@latest'})`. Throws an error. Second attempt, save the url as a `.js` file and include in the puppeteer block as `page.addScripTag({path: 'util.js'});`. Throws an error. Third attempt, saved a file called `main.js` with the following code `var util = require('util');`, then in terminal `browserify main.js -o bundle.js` and then included `bundle.js` in the `page.addScriptTag()`. Throws an error. Am I missing something here? Thanks again... – JdeMello Mar 22 '18 at 14:45
  • See Edit 2. Thanks! – JdeMello Mar 22 '18 at 15:02
  • Same error in all the cases ? And is that the only error you get? – Brahma Dev Mar 22 '18 at 15:11
  • See Edit 2, the error now is that it can't read the property `format`... `TypeError: Cannot read property 'format' of undefined at :5:55` – JdeMello Mar 22 '18 at 15:13
  • I've tried running the code and it works for me. Gives `Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket,Add to basket` – Brahma Dev Mar 22 '18 at 15:24
  • Thanks so much. I'll see what I can do here. Which piece of code did you copy? – JdeMello Mar 22 '18 at 15:26