22

I want to get the JSON data from a website I'm scraping with Puppeteer, but I can't figure how to get the body of the request back. Here's what I've tried:

const puppeteer = require('puppeteer')
const results = [];
(async () => {
    const browser = await puppeteer.launch({
        headless: false
    })
    const page = await browser.newPage()
    await page.goto("https://capuk.org/i-want-help/courses/cap-money-course/introduction", {
        waitUntil: 'networkidle2'
    });

    await page.type('#search-form > input[type="text"]', 'bd14ew')  
    await page.click('#search-form > input[type="submit"]')

    await page.on('response', response => {    
        if (response.url() == "https://capuk.org/ajax_search/capmoneycourses"){
            console.log('XHR response received'); 
            console.log(response.json()); 
        } 
    }); 
})()

This just returns a promise pending function. Any help would be great.

hardkoded
  • 18,915
  • 3
  • 52
  • 64
Rusty
  • 609
  • 1
  • 4
  • 19

1 Answers1

41

As response.json returns a promise we need to await it.

page.on('response', async (response) => {    
    if (response.url() == "https://capuk.org/ajax_search/capmoneycourses"){
        console.log('XHR response received'); 
        console.log(await response.json()); 
    } 
});
hardkoded
  • 18,915
  • 3
  • 52
  • 64
  • 1
    `await page.on` isn't necessary--`page.on` returns a page object for chaining more `.on` calls with, not a promise. For those seeing this thread in isolation, there's also [`page.waitForResponse`](https://stackoverflow.com/a/56839061/6243352) which lets you retrieve a response body right in an asynchronous chain. – ggorlen Nov 24 '20 at 17:13
  • Good catch @ggorlen! – hardkoded Nov 24 '20 at 20:47