Stack Overflow
Stack Overflow

Questions tagged [puppeteer]

Puppeteer is a Node.js library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

Puppeteer is a Node.js library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium. You can also use Puppeteer with Firefox Nightly (experimental support).

Puppeteer is mostly used for:

  1. Generate screenshots and PDFs of pages.
  2. Crawl an SPA and generate pre-rendered content (i.e. "SSR").
  3. Scrape content from websites.
  4. Automate form submission, UI testing, keyboard input, etc.
  5. Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  6. Capture a timeline trace of your site to help diagnose performance issues.

Resources:

Books:

7753 questions
40
votes
2 answers

page.evaluate Vs. Puppeteer $ methods

I'm interested in the differences of these two blocks of code. const $anchor = await page.$('a.buy-now'); const link = await $anchor.getProperty('href'); await $anchor.click(); await page.evaluate(() => { const $anchor =…
lpoulter
  • 503
  • 1
  • 4
  • 5
40
votes
5 answers

Set localstorage items before page loads in puppeteer?

We have some routing logic that kicks you to the homepage if you dont have a JWT_TOKEN set... I want to set this before the page loads/before the js is invoked. how do i do this ?
christoffee
  • 633
  • 1
  • 6
  • 15
38
votes
8 answers

How to use proxy in puppeteer and headless Chrome?

Please tell me how to properly use a proxy with a puppeteer and headless Chrome. My option does not work. const puppeteer = require('puppeteer'); (async () => { const argv = require('minimist')(process.argv.slice(2)); const browser = await…
Irina Kazhamiakina
  • 489
  • 1
  • 4
  • 4
38
votes
6 answers

Rendering WebGL image in headless chrome without a GPU

I'm trying to export an image rendered with WebGL on a linux server without a GPU. To do this I'm using headless Chrome however the exported image is black (example exported image, taking a screenshot of page shows its just canvas that is black). I…
James Hollingworth
  • 14,040
  • 12
  • 39
  • 57
36
votes
3 answers

Puppeteer page.evaluate querySelectorAll return empty objects

I am trying out Puppeteer. This is a sample code that you can run on: https://try-puppeteer.appspot.com/ The problem is this code is returning an array of empty…
Abdullah Alsigar
  • 1,236
  • 1
  • 12
  • 16
35
votes
12 answers

Could not find expected browser chrome locally

This Meteor code uses "puppeteer 8.0.0", "puppeteer-core 10.0.0", puppeteer-extra 3.1.18" and "puppeteer-extra-plugin-stealth 2.7.8", It gives this error: Error: Could not find expected browser (chrome) locally. Run npm install to download the…
Fred J.
  • 5,759
  • 10
  • 57
  • 106
34
votes
10 answers

How can I download images on a page using puppeteer?

I'm new to web scraping and want to download all images on a webpage using puppeteer: const puppeteer = require('puppeteer'); let scrape = async () => { // Actual Scraping goes Here... const browser = await puppeteer.launch({headless:…
supermario
  • 2,625
  • 4
  • 38
  • 58
34
votes
5 answers

Best way to intercept XHR request on page with Puppeteer and return mock response

I need to be able to intercept XHR requests on page loaded with Puppeteer and return mock responses in order to organize backendless testing for my web app. What's the best way to do this?
s.ermakovich
  • 2,641
  • 2
  • 26
  • 24
33
votes
4 answers

Puppeteer unable to run on Heroku

I deployed an app on heroku, and I added the Puppeteer Heroku buildpack. After a succesful redeployment, I tried to run it and it fails. Using heroku logs -t, I get this error message: 2018-09-07T13:16:10.870497+00:00 app[web.1]: Error: Failed to…
user1584421
  • 3,499
  • 11
  • 46
  • 86
33
votes
3 answers

Managing puppeteer for memory and performance

I'm using puppeteer for scraping some pages, but I'm curious about how to manage this in production for a node app. I'll be scraping up to 500,000 pages in a day, but these scrape jobs will happen at random intervals, so it's not a single queue that…
jeremywoertink
  • 2,281
  • 1
  • 23
  • 29
32
votes
2 answers

Why is puppeteer reporting "UnhandledPromiseRejectionWarning: Error: Navigation failed because browser has disconnected!"?

I've got a simple node.js script to capture screenshots of a few web pages. It appears I'm getting tripped up somewhere along the line with my use of async/await, but I can't figure out where. I'm currently using puppeteer v1.11.0. const puppeteer =…
Al R.
  • 2,430
  • 4
  • 28
  • 40
32
votes
10 answers

Inject jQuery into Puppeteer page

I'm trying to inject jQuery into my Puppeteer page because document.querySelector doesn't cut it for me: async function inject_jquery(page){ await page.evaluate(() => { var jq = document.createElement("script") jq.src =…
pguardiario
  • 53,827
  • 19
  • 119
  • 159
31
votes
8 answers

How can I check if selector exists in Puppeteer?

In Puppeteer, how can I check if, for example, #idProductType exists and if not, set producttype to ""? I tried many many things but it doesn't work. const urls = [myurls, ...] const productsList = []; for (let i = 0; i < urls.length; i++) { …
Cyri1
  • 389
  • 1
  • 3
  • 9
31
votes
1 answer

How to type a text in input box in puppeteer

I need to know how can we type a string in input box using puppeteer. I know it can be done like this: await page.type('input[name=pickup]', 'test comment', {delay: 200}) But what if the input box does not have a name or id, and instead it has a…
Vinirdishtith Rana
  • 793
  • 1
  • 7
  • 13
30
votes
1 answer

error while loading shared libraries: libgbm.so.1: cannot open shared object file: Puppeteer in Nodejs on AWS EC2 instance

I'm using a Puppeteer module to scrape some data using Node.js. It was working fine on my Macbook but when I deployed the build on an AWS EC2 instance and hit the route that was scraping data it started throwing an error: error while loading shared…
ImFarhad
  • 2,669
  • 2
  • 18
  • 30
1 2 3
…
99 100