Stack Overflow
Stack Overflow

Questions tagged [pyppeteer]

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Pyppeteer is mostly used for:

  1. Generate screenshots and PDFs of pages.
  2. Crawl an SPA and generate pre-rendered content (i.e. "SSR").
  3. Scrape content from websites.
  4. Automate form submission, UI testing, keyboard input, etc.
  5. Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  6. Capture a timeline trace of your site to help diagnose performance issues.

Resources:

Differences from puppeteer

185 questions
1
vote
1 answer

requests_html render method continues to result in timeout

I'm not sure if it's the setting of my computer (Windows 7, Python3.8, just recently installed requests_html), but no matter which site I choose, the render method is resulting in timeout error I have played around with timeout parameter, but no…
MasayoMusic
  • 594
  • 1
  • 6
  • 24
1
vote
0 answers

Pyppeteer query page contents after timeout?

After navigating to a page, I would like to wait for either: The page to load, OR A timeout (20 seconds) However, after catching a TimeoutError, it reports that the original page has crashed and I cannot retrieve the contents. Is it possible to…
rosstex
  • 773
  • 1
  • 9
  • 23
1
vote
1 answer

How to download a csv using pyppeteer?

I am new to pyppeteer and try to download a csv. Unfortunately, without success. Suppose I want to download the following csv: 'https://people.sc.fsu.edu/~jburkardt/data/csv/addresses.csv'. Simply using await…
Patrick Balada
  • 1,330
  • 1
  • 18
  • 37
1
vote
1 answer

Puppeteer: stop waiting for a slow page once a special element has already appeared

I need to visit some independent URLs where the elements of the pages are loaded very slowly. It usually takes several minutes to make the entire page completely loaded. However, only a small portion of this page is useful. The useful part can be…
Bo Wang
  • 55
  • 7
1
vote
1 answer

How can I properly send a POST request to a Website in Pyppeteer

I am attempting to write a bot in pyppeteer. What I am attempting to do with my code is send a POST request to a website with specific postData Add_url = f"https://www.website.com/shop/{productID}/add.json" await…
Floaters
  • 33
  • 1
  • 4
1
vote
2 answers

How to disable logging in pyppeteer

I'm using pyppeteer to take screenshots of images to a make a pdf but pyppeteer auto logs everything I take a screenshot at and because of server limitations and the logs is written to a file the logs are crashing my server. Is there any way to…
Juan Carlos
  • 578
  • 5
  • 22
1
vote
2 answers

Pyppeteer. Chromium browser stops loading pages after set period of time

I'm trying to write a program using Pyppeteer and asyncio that would take screenshots of different websites. But every time I ran the program after about 20 seconds an error occured (Runtime Error: Session closed. Most likely the page has been…
timafefi
  • 13
  • 4
1
vote
1 answer

How to access source code from within Docker build?

I'm trying to build a Github Action that's going to take a screenshot of the Github Pages website (built with Jekyll) and allow the user to upload it as an artifact. For my convienience and since installing Pyppeteer is non-trivial, I wrapped the…
d33tah
  • 10,999
  • 13
  • 68
  • 158
1
vote
2 answers

'coroutine' object has no attribute get || pyppeteer

In python, using pyppeteer, I am opening a webpage and running a JS script in its console and trying to capture the result in a variable but I am getting the following error. Traceback (most recent call last): File…
Mahesh
  • 1,117
  • 2
  • 23
  • 42
1
vote
1 answer

How to wait for Recaptcha to load in Puppeteer/Pyppeteer?

I want to scrape a website and I'm having difficulties with the Recaptcha. I've already figured out a way to solve it but before that method starts I have to make sure Recaptcha is fully loaded, which is what I'm stuck at. I've tried…
Leo
  • 370
  • 7
  • 12
1
vote
0 answers

How to wait for a Recaptcha to load in Pyppeteer?

I want to scrape a website and I'm having difficulties with the Recaptcha. I've already figured out a way to solve it but before that method starts I have to make sure Recaptcha is fully loaded, which is what I'm stuck at. I've tried…
Leo
  • 370
  • 7
  • 12
1
vote
1 answer

Pyppeteer crushes after 20 seconds with pyppeteer.errors.NetworkError

During usage of pyppeteer for controlling the Chromium I have been receiving an error approximately after 20 seconds of work: pyppeteer.errors.NetworkError: Protocol Error (Runtime.callFunctionOn): Session closed. Most likely the page has been…
Petr Matuska
  • 553
  • 5
  • 15
1
vote
0 answers

pyppeteer - How to execute a javascript function that is inside the page?

I have this element: Net Err. Please refresh I…
Joe
  • 791
  • 1
  • 9
  • 24
1
vote
1 answer

Pyppeteer behaving differently on Linux and Windows

I have pyppeteer code that browses around. Let's assume it only clicks on a tags. It runs fine on my local Windows machine, but breaks whenever I run it remotely on a Linux server. Same conda env, same code. The relevant part of my code, simplified,…
YardenR
  • 301
  • 2
  • 13
1
vote
0 answers

With pyppeteer how to make a request with client cert?

I have the need to reach a page using client certificate with pyppeteer, i have seen that its possible with puppeteer itself here, but my attempts lead to an invalid state errror with python3. Im not sure whether my attempt is entirely correct,…
user122508
  • 76
  • 6
1 2 3
…
12 13