How do I intercept all f the page traffic in a scrollable website on playwright?

Question

So I am using the following code:

import { firefox } from 'playwright';
// domain
let domain = 'https://www.reddit.com/'

// create a new page
const page = await browser.newPage();
// set routes
await page.route('**', async route => {
  const response = await route.fetch();
  await route.fulfill({ response });
});
// go to the domain
await page.goto(domain);

When run it I can see the traffic go trough my route, but after a seomt time as I start to scroll down the reddit page I stop seeing it. I can see traffic comming in on the dev tools, but it does not get intecepted by my route. I think it might be intercepting all of the traffic only to load the page, but not anyother requests after that. How can I intercept all of the traffic comming from and to the page. Thank you ^^

I tried other scrollabel web apps such a tiktok with the same results. I want to be able to route all of the traffic coming from my page.

Answer 1

You need to handle the request event instead of the fetch event:

  import { firefox } from 'playwright';
  const browser = await firefox.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  // Set up the request interception
  await page.route('**', async (route) => {
    const request = route.request();
    console.log('Intercepted request:', request.url());
    await route.continue();
  });

  // Navigate to the website
  await page.goto('https://www.reddit.com/');

  // Scroll down to trigger additional requests
  await page.evaluate(() => {
    window.scrollTo(0, document.body.scrollHeight);
  });

  // Wait for a while to allow more requests to be intercepted
  await page.waitForTimeout(3000);

  // Close the browser
  await browser.close();

Answer 2

Monitoring Network

With playwright, you can monitor all the Requests and Responses on a page:

// Subscribe to 'request' and 'response' events.
page.on('request', request => console.log('>>', request.method(), request.url()));
page.on('response', response => console.log('<<', response.status(), response.url()));

// Navigate to the website
  await page.goto('https://www.reddit.com/');



// Scroll down to trigger additional requests
    await page.evaluate(() => {
        window.scrollTo(0, document.body.scrollHeight);
    });