Stack Overflow
Stack Overflow

Questions tagged [pyppeteer]

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Pyppeteer is mostly used for:

  1. Generate screenshots and PDFs of pages.
  2. Crawl an SPA and generate pre-rendered content (i.e. "SSR").
  3. Scrape content from websites.
  4. Automate form submission, UI testing, keyboard input, etc.
  5. Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  6. Capture a timeline trace of your site to help diagnose performance issues.

Resources:

Differences from puppeteer

185 questions
2
votes
1 answer

Handling multiple tabs in using pyppeteer

I am using peppeteer to retrieve website data and want to open another new tab (second tab) to retrieve more detailed data. The new tab will be closed after the specified data is retrieved and switch back to the first tab. I had tried to open…
Kester
  • 283
  • 2
  • 5
  • 10
2
votes
1 answer

Script throws some error at some point within the execution

I've created a script in python using pyppeteer to collect the links of different posts from a webpage and then parse the title of each post by going in their target page reusing those collected links. Although the content are static, I like to know…
robots.txt
  • 96
  • 2
  • 10
  • 36
2
votes
1 answer

How to set up Accept-Encoding to gzip in Python pyppeteer and print pyppeteer headers?

how to set headers for pyppeteer for example:Accept-Encoding: gzip how to print pyppeteer headers in python. i know java const response = await page.goto('https://example.org/') console.log(response.headers) result { date: 'Sun, 29 Oct 2017…
xin.chen
  • 964
  • 2
  • 8
  • 24
2
votes
3 answers

select a button by its text in Pyppeteer (Python Puppeteer)

I want to click the following button using pyppeteer I am trying to do it using Jquery like suggested here: btn = await page.querySelector('button[text=\'Text here\']') or here: btn = await…
ben
  • 1,064
  • 3
  • 15
  • 29
2
votes
2 answers

Ideal way of creating for loops using pyppeteer

I've written a script in python in combination with pyppeteer to scrape the names and its phone numbers of different coffe shops from a webpage. Although the way I tried below serves the purpose, the script looks real messy. What is the ideal way of…
robots.txt
  • 96
  • 2
  • 10
  • 36
2
votes
0 answers

pyppeteer.errors.BrowserError: Failed to connect to browser port

I have problems when using requests-html package on Python 3.6.5, Ubuntu 16.04(x64). To be more specific, the last line of from requests_html import HTMLSession session = HTMLSession() r =…
user9928000
  • 21
  • 2
1
vote
1 answer

RuntimeError: Event loop is closed. Pyppeteer

I am trying to write a program that handles HTTP requests and takes a screenshot of the links in the query parameters. I want to make it so that the browser stays open and requests create tabs in order to save RAM. The problem is that the first…
mfkrg
  • 13
  • 2
1
vote
1 answer

PermissionError Multiprocessing argument pyppeteer.Page

PermissionError Multiprocessing argument pyppeteer.Page successful but inefficient import asyncio from pyppeteer import launch from multiprocessing import Process async def f(x): print("async def f(x,page):",x) …
금밈미
  • 21
  • 2
1
vote
0 answers

Pyppeteer will stop working at the moment of opening the browser

I am using python version 3.9. I have a code that repeats the same tests using a "While" loop. async def pars(): site_ad = "my_link" msg = '' new_mass_for_mes = [] try: launcher.defaultArgs().remove("--enable-automation") …
The person with the question
  • 93
  • 1
  • 10
1
vote
1 answer

pyppeteer.errors.ElementHandleError: Evaluation failed: SyntaxError: Unexpected token return

I want to scrape the x and y axis of a highcharts graph. As shown here: https://www.highcharts.com/demo/line-basic I´m using html_requests which uses pypeteer to send JavaScript. chart = r.get("a.com") script= """return…
MaxFrost
  • 91
  • 6
1
vote
1 answer

nbconvert failed: No suitable chromium executable found on the system. Please use '--allow-chromium-download' to allow downloading one

Getting error as "nbconvert failed: No suitable chromium executable found on the system. Please use '--allow-chromium-download' to allow downloading one." while trying to download jupyter notebook file (.ipynb) as a PDF file
Girish Shenoy
  • 429
  • 1
  • 4
  • 10
1
vote
1 answer

Is there a way to sleep certain chunks of python code similar to time.sleep() without sleeping the entire script

Basically I'm using pyppeteer to connect to an existing browser connection which requires me to periodically time.sleep() the thread in order for the browser to behave normally (using asyncio.sleep() still causes dynamic HTML websites to behave…
Jimmy J.K.
  • 25
  • 6
1
vote
0 answers

How do I get the data inside of React props while Web scraping with Python?

Summary I am trying to scrape the website Quizlet using Pyppeteer (The Python Port of Puppeteer), however, I am running into a problem trying to obtain the prop data from the components, while scraping as the site uses React. I looked at the prop…
Tosin Kuye
  • 11
  • 1
  • 2
1
vote
2 answers

Is it possible to get pdf page using pyppeteer?

import asyncio import pyppeteer import logging from pyppeteer import launch pyppeteer.DEBUG = True for name in logging.root.manager.loggerDict: logging.getLogger(name).disabled = True async def main(): browser = await launch(headless =…
Alex
  • 1,047
  • 8
  • 21
1
vote
1 answer

Extract full HTML of a website by using pyppeteer in python

I'm using the below code to extract full HTML: cont = await page1.content() The website I intend to extract from is: https://www.mohmal.com/en which is a website to make temporary email accounts. The exact thing I want to do is reading the content…
فرهاد سلیمانی
  • 11
  • 2
1 2 3
…
12 13