Stack Overflow
Stack Overflow

Questions tagged [puppeteer]

Puppeteer is a Node.js library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

Puppeteer is a Node.js library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium. You can also use Puppeteer with Firefox Nightly (experimental support).

Puppeteer is mostly used for:

  1. Generate screenshots and PDFs of pages.
  2. Crawl an SPA and generate pre-rendered content (i.e. "SSR").
  3. Scrape content from websites.
  4. Automate form submission, UI testing, keyboard input, etc.
  5. Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  6. Capture a timeline trace of your site to help diagnose performance issues.

Resources:

Books:

7753 questions
24
votes
6 answers

How to await inside setInterval in JS?

I have a code segment that looks like this: async function autoScroll(page, maxDate = null) { await page.evaluate(async () => { await new Promise(async (resolve, reject) => { try { const scrollHeight =…
tinker
  • 2,884
  • 8
  • 23
  • 35
24
votes
3 answers

Passing multiple cookies to puppeteer

By doing what user Md. Abu Taher suggested, i used a plugin called EditThisCookie to download the cookies from my browser. The exported cookies are in JSON format, in fact it is an array of objects. Is it possible to pass this array as a parameter…
user1584421
  • 3,499
  • 11
  • 46
  • 86
24
votes
11 answers

Puppeteer waitForSelector on multiple selectors

I have Puppeteer controlling a website with a lookup form that can either return a result or a "No records found" message. How can I tell which was returned? waitForSelector seems to wait for only one at a time, while waitForNavigation doesn't seem…
Jon Wilson
  • 726
  • 1
  • 8
  • 23
24
votes
7 answers

puppeteer - how to set download location

I was able to successfully download a file with puppeteer, but it was just saving it to my /Downloads folder. I've been looking around and can't find anything in the api or forums to set this location. My downloads are basically just go going to the…
A. L
  • 11,695
  • 23
  • 85
  • 163
24
votes
6 answers

Wait for element to appear when using Puppeteer

I wonder if there's a similar way as in Selenium to wait for text to appear for a particular element. I've tried something like this, but it doesn't seem to wait: await page.waitForSelector('.count', {visible: true});
elena
  • 3,740
  • 5
  • 27
  • 38
24
votes
5 answers

How to handle popups in puppeteer

how to handle the popup and access the popup to do some operations on it. const puppeteer = require('puppeteer'); async function run() { const browser = await puppeteer.launch(); const page = await browser.newPage(); await…
vinay kumar
  • 555
  • 1
  • 6
  • 16
23
votes
3 answers

Want to scrape table using Puppeteer. How can I get all rows, iterate through rows, and then get "td's" for each row?

I have Puppeteer setup, and I was able get all of the rows using: let rows = await page.$$eval('#myTable tr', row => row); Now I want for each row to get "td's" and then get the innerText from those. Basically I want to do this: var tds =…
mirceah13
  • 275
  • 1
  • 2
  • 9
23
votes
3 answers

How to use puppeteer to dump WebSocket data

I want to get websocket data in this page https://upbit.com/exchange?code=CRIX.UPBIT.KRW-BTC, its websocket URL is dynamic and only valid during the first connection, the second time you connect to it it will not send data anymore. So I wonder that…
soulmachine
  • 3,917
  • 4
  • 46
  • 56
23
votes
6 answers

Retrieving JavaScript Rendered HTML with Puppeteer

I am attempting to scrape the html from this NCBI.gov page. I need to include the #see-all URL fragment so that I am guaranteed to get the searchpage instead of retrieving the HTML from an incorrect gene page…
Sir_Zorg
  • 331
  • 1
  • 2
  • 5
22
votes
2 answers

Puppeteer error: An `executablePath` or `channel` must be specified for `puppeteer-core`

I'm trying to make a simple Puppeteer project. My current code is just a test. It doesn't work though. import bypass from './captcha/captchaBypasser.js'; import {createRequire} from "module"; const require = createRequire(import.meta.url); const…
Giboos Montażysta
  • 247
  • 1
  • 2
  • 8
22
votes
1 answer

How to get body / json response from XHR request with Puppeteer

I want to get the JSON data from a website I'm scraping with Puppeteer, but I can't figure how to get the body of the request back. Here's what I've tried: const puppeteer = require('puppeteer') const results = []; (async () => { const browser =…
Rusty
  • 609
  • 1
  • 4
  • 19
22
votes
4 answers

Node console.log on large array shows "... 86 more items"

I'm new to puppeteer. I used to have PhantomJS and CasperJS but while setting a newer server (freebsd 12) found out that support for PhantomJS is gone and CasperJS gives me segmentation faults. I was able to port my applications to puppeteer just…
John Ralston
  • 233
  • 1
  • 2
  • 5
22
votes
3 answers

Is it possible to modify an element in the DOM with Puppeteer before creating a screenshot?

I ran into an issue where I nave a fairly simple Node process that captures a screenshot. Is it possible to change the innerText of an HTML element using Puppeteer, just before the screen capture is acquired? I have had success with using Puppeteer…
tamak
  • 1,541
  • 2
  • 19
  • 39
21
votes
3 answers

puppeteer not working on vps but running locally

I wrote a little puppeteer program that let me log into twitter and check a few things. Locally on mac OS Catalina, it is working but on VPS ubuntu 18.04 lts not working. And shows me a log at the…
programmer
  • 550
  • 2
  • 4
  • 25
21
votes
2 answers

How to select elements within an iframe element in Puppeteer

Since ESPN does not provide an API, I am trying to use Puppeteer to scrape data about my fantasy football league. However, I am having a hard time trying to login using puppeteer due to the login form being nested with an iframe element. I have gone…
SwapnikK
  • 213
  • 1
  • 2
  • 4
1 2 3
…
99 100