Puppeteer Page.Screencast click on page

Question

I'm using the screencast from CDP to transmit the screen of the Puppeteer page to my website and I want to add a new function, where you click on the transmission the click on the page happens too.

The screencast code and socket connections are all in order, the problem is the click coordinates that seem to be wrong and I can't sync them

To put the image on my website I am receiving the screencast frame and sending it via socket:

await session.send('Page.startScreencast')
session.on('Page.screencastFrame', ({ data: base64, sessionId }) => {
   socket.emit('screencastFrame', { base64 }) // sending base64 image
   session.send('Page.screencastFrameAck', { sessionId }).catch(() => { });
})

on my website the image (800x604 default) is shown and the click event:

<img
   src={`data:image/png;base64, ${base64}`}
   onClick={({ screenX, screenY }) => {
      socket.emit('click', { coords: { x: screenX, y: screenY } })
   }} />

to handle the click event I do this:

socket.on('click', ({coords}) => {
   await page.evaluate(({ x, y }) => {
      const element = document.elementFromPoint(x, y);
      console.log((element as HTMLElement))
      element.click() 
   }, coords))
})

As I said earlier, clicks are being performed in the wrong places I don't know a solution for this

score 0 · Answer 1 · answered Jan 14 '21 at 09:51

You need to use clientX and clientY instead of screenX and screenY.

So changing your code to this one belove will solve the problem.

<img
   src={`data:image/png;base64, ${base64}`}
   onClick={({ screenX, screenY }) => {
      socket.emit('click', { coords: { x: clientX, y: clientY } })
   }} />

Puppeteer Page.Screencast click on page

1 Answers1