Detecting POST response arrival after clicking with Puppeteer

Question

I'm trying to capture the information from a table but this table has a pagination button ("next , prev"). When I click the next button, no navigation occurs; it just creates a POST request that returns a new HTML table.

When I click the next button that causes a POST (not navigation), how can I wait to this POST to finish before trying to capture the data again from the next page?

Maybe I can detect some changes in the table element, but I don't know how.

What is the best approach for this problem?

Right now I'm doing this:

while (await page.$(NEXT_BUTTON_SELECTOR) !== null) {
  await page.click(NEXT_BUTTON_SELECTOR);
  await page.waitFor(2 * 1000);
  pageTableArray = getData();
}

but I'm not convinced that this is a good way to do it.

Does this answer your question? [puppeteer: Access JSON response of a specific request as in the network tab of DevTools](https://stackoverflow.com/questions/56514308/puppeteer-access-json-response-of-a-specific-request-as-in-the-network-tab-of-d) — ggorlen, Nov 24 '20 at 06:17

score 1 · Answer 1 · answered May 30 '18 at 07:02

1

If await page.click(NEXT_BUTTON_SELECTOR); is causing a pageload to happen then whatever happens next in the script is lost. To get around that you'd have to do:

page.click(NEXT_BUTTON_SELECTOR); // Notice no `await`
await page.waitForNavigation();

Check out more about that API from the docs page here!

answered May 30 '18 at 07:02

browserless

2,090
16
16

OP specified that no navigation occurs, so `waitForNavigation` won't help. – ggorlen Nov 24 '20 at 04:27

score 1 · Answer 2 · edited Nov 24 '20 at 04:28

1

You can use event requestfinished to capture data.

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', req => {
    console.log('request:', req.url())
    req.continue();
});
page.on('requestfinished', (req) => {
    console.log('finished:', req.url())
});
page.on('requestfailed', (req) => {
    console.log('failed:', req.url())
})
await page.goto(url);
await page.click(selector);

edited Nov 24 '20 at 04:28

ggorlen

44,755
7
76
106

answered May 30 '18 at 08:19

Evan Zhang

11
2

This helps by giving you access to the response in a callback, but after clicking, if you want to access data or at least await the request completion in the promise chain that fires the click, try [`page.waitForResponse`](https://stackoverflow.com/a/56514637/6243352). – ggorlen Nov 24 '20 at 06:16

Detecting POST response arrival after clicking with Puppeteer

2 Answers2