Stack Overflow
Stack Overflow

Questions tagged [pyppeteer]

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Pyppeteer is mostly used for:

  1. Generate screenshots and PDFs of pages.
  2. Crawl an SPA and generate pre-rendered content (i.e. "SSR").
  3. Scrape content from websites.
  4. Automate form submission, UI testing, keyboard input, etc.
  5. Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  6. Capture a timeline trace of your site to help diagnose performance issues.

Resources:

Differences from puppeteer

185 questions
0
votes
0 answers

How do I reference an ::after tag in order to click it using puppeteer/pyppeteer?

I am trying to automate some tasks at work and this requires getting page speed insights from pingdom for a lot of pages. I have no budget to purchase premium so I'm writing a python script myself. I need to click 'North America - USA - San…
Weeoowee
  • 25
  • 5
0
votes
1 answer

How to fetch a url asynchronously with pyppeteer(One browser many tabs)

I want my script to Open say 3 tabs Asynchronously fetch a url(same for each tab) Save the response Sleep for 4 seconds Parse through the response with regex(I tried BeautifulSoup but its too slow) and return a token Loop through several times…
Muhammad Luay
  • 23
  • 1
  • 8
0
votes
1 answer

PYppeteer times out even though the page has already loaded (no spinner, no network activity)

I have the following code that automates the login to a particular site, in order to grab some value that is behind a login screen. #!/usr/bin/python3 import asyncio from pyppeteer import launch browser = await launch( {'headless': True} ); page =…
Alexei Masterov
  • 382
  • 2
  • 10
0
votes
1 answer

Returning values to python from page.evaluate in Pyppeteer (Python Puppeteer)

I'm using Pypeteer to screen scrape a page and I have the following code which selects various elements. foo = await page.evaluate(""" var name = document.querySelectorAll("h2")[0].innerText var balance =…
Christian Papathanasiou
  • 321
  • 1
  • 3
  • 12
0
votes
1 answer

Pyppeteer - error when running page.content() after clicking on a link

A beginner in JS/Html, I'm trying to fetch the content of a page after clicking on a link, through Pyppeteer0.2.5 (Python3.6.9/Chromium 87.0.4280.66) using the following code: import asyncio, pyppeteer, time from pyppeteer import launch async def…
The Authors
  • 131
  • 5
0
votes
1 answer

Heroku error after deploying (No module named '_tkinter')

I have an web app deployed on heroku (using Python), I am using now Pyppeteer to make some automatic jobs in a web site. After deploying I am getting this issue: Request Method: GET 3.1.4 ModuleNotFoundError No module named '_tkinter' …
efr
  • 301
  • 3
  • 10
0
votes
1 answer

How to save Network.webSocketFrameReceived to Python dictionary. Using Pyppeteer library for UI test

I use Pyppeteer library because there is a Chrome Dev Tools protocol and I can receive webSocketFrameReceived after sending a request in UI test. I was able to print the socket response to the terminal, but that's not what I want. I need, depending…
Anatolii Olesh
  • 44
  • 7
0
votes
0 answers

pyppeteer command altering the size of window [bug??]

I'm developing a program which needs the use of pyppeer to control chromium instance. I'm coming into trouble when my go back button is used, the size of the window is altered. How do I prevent this from happening? Here is a shortened version of my…
TheOuz
  • 47
  • 7
0
votes
1 answer

Pyppeteer session crash or Timeout Exceeded

For the purpose of scraping binance.com, I use the library pyppeteer to render the web page and get clean html code instead of javascript code. My problem is this: the first time the session works properly on the remote Ubuntu 20.04 server, but when…
Egor Markin
  • 13
  • 4
0
votes
1 answer

How to get search result URLs by Pyppeteer?

I am trying to scrape the searching result's url by Pyppeteer in my Python program, but it doesn't work... And here is my code: import asyncio from pyppeteer import launch URL = 'https://hk.appledaily.com/search/apple' async def main(): …
7777777777 die
  • 51
  • 3
0
votes
1 answer

pyppeteer headless results in bot detection, while not headless is okay

I'm experimenting with Pyppeteer and am unsure why headless is resulting in bot detection, while headless = False, isn't. Here is my code (please note proxy-sever args will need to be removed or replaced with your proxy server): I'm assuming there…
MasayoMusic
  • 594
  • 1
  • 6
  • 24
0
votes
1 answer

Scraping dynamic content from website in near-realtime

I’m trying to implement a web scraper scraping dynamically updated content from a website in near-realtime. Let’s take https://www.timeanddate.com/worldclock/ as an example and assume I want to continuously get the current time at my home…
Cake
  • 177
  • 2
  • 9
0
votes
1 answer

How to open url in new tab with pyppeteer?

With nodejs and puppeteer it looks like: const puppeteer = require('puppeteer'); const browser = await puppeteer.launch(); const page1 = await browser.newPage(); await page1.goto('https://google.com'); …
Qwentor
  • 31
  • 1
  • 4
0
votes
0 answers

fill form of popup or prompted window in pyppeteer python

I am using pyppeteer in python to do some automation.Everything was fine until i got one form in popup window which i am trying to fill, i am not able to take control of that popup/prompted form. I tried to get control of this popup by using…
user3420561
  • 188
  • 1
  • 2
  • 20
0
votes
1 answer

How to change operating system detected by headless Chrome?

I use pyppeteer. But when I go to https://ipleak.net it detects my real OS. I try to load this scrypt with async with aiofiles.open(os.path.join(BASEDIR, 'preload.js'), mode='r') as f: preloadFile = await f.read() # code for opening browser…
Qwentor
  • 31
  • 1
  • 4
1 2 3
…
12 13