3

For the context, I am developing a synthetic monitoring tool using Nodejs and puppeteer. For each step of a defined scenario, I capture a screenshot, a waterfall and performance metrics.

My problem is on the waterfall, I previously used puppeter-har but this package is not able to capture request outside of a navigation. Therefore I use this piece of code to capture all interesting requests :

const {harFromMessages} = require('chrome-har');
// Event types to observe for waterfall saving (probably overkill, I just set all events of Page and Network)
const observe = [
  'Page.domContentEventFired',
  'Page.fileChooserOpened',
  'Page.frameAttached',
  'Page.frameDetached',
  'Page.frameNavigated',
  'Page.interstitialHidden',
  'Page.interstitialShown',
  'Page.javascriptDialogClosed',
  'Page.javascriptDialogOpening',
  'Page.lifecycleEvent',
  'Page.loadEventFired',
  'Page.windowOpen',
  'Page.frameClearedScheduledNavigation',
  'Page.frameScheduledNavigation',
  'Page.compilationCacheProduced',
  'Page.downloadProgress',
  'Page.downloadWillBegin',
  'Page.frameRequestedNavigation',
  'Page.frameResized',
  'Page.frameStartedLoading',
  'Page.frameStoppedLoading',
  'Page.navigatedWithinDocument',
  'Page.screencastFrame',
  'Page.screencastVisibilityChanged',
  'Network.dataReceived',
  'Network.eventSourceMessageReceived',
  'Network.loadingFailed',
  'Network.loadingFinished',
  'Network.requestServedFromCache',
  'Network.requestWillBeSent',
  'Network.responseReceived',
  'Network.webSocketClosed',
  'Network.webSocketCreated',
  'Network.webSocketFrameError',
  'Network.webSocketFrameReceived',
  'Network.webSocketFrameSent',
  'Network.webSocketHandshakeResponseReceived',
  'Network.webSocketWillSendHandshakeRequest',
  'Network.requestWillBeSentExtraInfo',
  'Network.resourceChangedPriority',
  'Network.responseReceivedExtraInfo',
  'Network.signedExchangeReceived',
  'Network.requestIntercepted'
];

At the start of the step :

// list of events for converting to HAR
  const events = [];

  client = await page.target().createCDPSession();
  await client.send('Page.enable');
  await client.send('Network.enable');
  observe.forEach(method => {
    client.on(method, params => {
      events.push({ method, params });
    });
  });

At the end of the step :

waterfall = await harFromMessages(events);

It works good for navigation events, and also for navigation inside a web application. However, the web application I try to monitor has iframes with the main content. I would like to see the iframes requests into my waterfall.

So a few question :

  • Why is Network.responseReceived or any other event doesn't capture this requests ?
  • Is it possible to capture such requests ?

So far I've red the devtool protocol documentation, nothing I could use. The closest to my problem I found is this question : How can I receive events for an embedded iframe using Chrome Devtools Protocol?

My guess is, I have to enable the Network for each iframe I may encounter. I didn't found any way to do this. If there is a way to do it with devtool protocol, I should have no problem to implement it with nodsjs and puppeteer.

Thansk for your insights !

EDIT 18/08 :

After more searching on the subject, mostly Out-of-process iframes, lots of people on the internet point to that response : https://bugs.chromium.org/p/chromium/issues/detail?id=924937#c13

The answer is question states :

Note that the easiest workaround is the --disable-features flag.

That said, to work with out-of-process iframes over DevTools protocol, you need to use Target [1] domain:

  • Call Target.setAutoAttach with flatten=true;
  • You'll receive Target.attachedToTarget event with a sessionId for the iframe;
  • Treat that session as a separate "page" in chrome-remote-interface. Send separate protocol messages with additional sessionId field:
    {id: 3, sessionId: "", method: "Runtime.enable", params: {}}
  • You'll get responses and events with the same "sessionId" field which means they are coming from that frame. For example:
    {sessionId: "", method: "Runtime.consoleAPICalled", params: {...}}

However I'm still not able to implement it.

I'm trying this, mostly based on puppeteer :

  const events = [];
  const targets = await browser.targets();
  const nbTargets = targets.length;
  for(var i=0;i<nbTargets;i++){
    console.log(targets[i].type());
    if (targets[i].type() === 'page') {
      client = await targets[i].createCDPSession();

      await client.send("Target.setAutoAttach", {
        autoAttach: true,
        flatten: true,
        windowOpen: true,
        waitForDebuggerOnStart: false // is set to false in pptr
      })

      await client.send('Page.enable');
      await client.send('Network.enable');

      observeTest.forEach(method => {
        client.on(method, params => {
          events.push({ method, params });
        });
      });
    }
  };

But I still don't have my expected output for the navigation in a web application inside an iframe.

However I am able to capture all the requests during the step where the iframe is loaded. What I miss are requests that happened outside of a proper navigation.

Does anyone has an idea about the integration into puppeteer of that chromium response above ? Thanks !

Félix
  • 134
  • 8

1 Answers1

1

I was looking on the wrong side all this time. The chrome network events are correctly captured, as I would have seen earlier if I checked the "events" variable earlier. The problem comes from the "chrome-har" package that I use on :

waterfall = await harFromMessages(events);

The page expects the page and iframe main events to be present in the same batch of event than the requests. Otherwise the request "can't be mapped to any page at the moment".

The steps of my scenario being sometimes a navigation in the same web application (=no navigation event), I didn't have these events and chrome-har couldn't map the requests and therefore sent an empty .har

Hope it can help someone else, I messed up the debugging on this one...

Félix
  • 134
  • 8