13

I'd like to directly get the response of the last HTTP request shown in the screenshot.

The current puppeteer code is shown below. Could anybody show me how to modify it so that it will get the JSON response directly from the browser? Thanks.

const puppeteer = require('puppeteer');

(async () => {
    //  const browser = await puppeteer.launch();
    const browser = await puppeteer.launch({
        headless: false
        , args: ['--user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3312.0 Safari/537.36"']
    });
    const page = await browser.newPage();
    await page.goto('https://www.genecards.org/cgi-bin/carddisp.pl?gene=BSCL2');
    await page.goto('https://www.genecards.org/cgi-bin/carddisp.pl?gene=BSCL2');

    const linkHandlers = await page.$x("//div[@id='enhancers']//a[@data-track-event='Table See-All']");
    if (linkHandlers.length > 0) {
        await linkHandlers[0].click();
    } else {
        throw new Error("Link not found");
    }

    const html = await page.content()
    //await browser.close();
    console.log(html)
})();

enter image description here

Thomas Dondorf
  • 23,416
  • 6
  • 84
  • 105
user1424739
  • 11,937
  • 17
  • 63
  • 152

1 Answers1

19

You can use page.waitForResponse to wait for the response and response.json to parse the response as JSON.

Code

Replace the await linkHandlers[0].click(); part by this:

const [response] = await Promise.all([
    page.waitForResponse(response => response.url().includes('/gene/api/data/Enhancers')),
    linkHandlers[0].click()
]);
const dataObj = await response.json();
console.log(dataObj);

This will first wait for the response (while in parallel making the click). After the response is detected the response is parsed as JSON. To get the response result as plain text (instead of parsing it), you can use response.text()

Thomas Dondorf
  • 23,416
  • 6
  • 84
  • 105
  • Thanks. What is the square bracket around `response` mean? – user1424739 Jun 09 '19 at 13:10
  • 1
    @user1424739 That's [destructuring assignment](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment). The `Promise.all` will return both results as an array (with two elements), but my code only takes the first result (as this is the only one we need). – Thomas Dondorf Jun 09 '19 at 14:00