Stack Overflow
Stack Overflow

Questions tagged [pyppeteer]

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation library.

Pyppeteer is mostly used for:

  1. Generate screenshots and PDFs of pages.
  2. Crawl an SPA and generate pre-rendered content (i.e. "SSR").
  3. Scrape content from websites.
  4. Automate form submission, UI testing, keyboard input, etc.
  5. Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  6. Capture a timeline trace of your site to help diagnose performance issues.

Resources:

Differences from puppeteer

185 questions
1
vote
1 answer

The connection closed when code running. which use pyppeteer to crawl a web

I make a test that crawler taobao.com by use pyppeteer. the taobao.com have identify code, which is the slider button, so I have add some method in code. but the error happend when the code running. the error info like that: 2018-11-30 18:15:32…
Yu Chen
  • 49
  • 4
1
vote
1 answer

Can't create a loop to fetch all the titles from a webpage

I've written a script in python in combination with pyppeteer to scrape the titles and links to the titles of different posts from a webpage. The thing is when I run my script, it can parse the first title and the link of the first post there. My…
robots.txt
  • 96
  • 2
  • 10
  • 36
1
vote
1 answer

Set timeout time works not in all cases in Pyppeteer

I set timeout time in Pyppeteer to 60 sec, but it doesn't work for all pages. Here is my code: options = {'timeout': 60 * 1000} response = await self.page.goto(url, options) I try to load this URL with a timeout: http://www.google.com:81 But after…
amarynets
  • 1,765
  • 10
  • 27
1
vote
1 answer

pyppeteer setCookie issue

When I use pyppeteer with cookies, the below exception is raised File "test.py", line 33, in asyncio.get_event_loop().run_until_complete(main()) File "/usr/local/python3.6/lib/python3.6/asyncio/base_events.py", line 468, in…
Delphi
  • 31
  • 1
  • 4
1
vote
0 answers

pyppeteer script crashing after pressing Ctrl+C

Consider the following code: import asyncio import pyppeteer async def hi(): ret = await pyppeteer.launch() await asyncio.sleep(10) return ret browser = asyncio.get_event_loop().run_until_complete(hi()) If I run it, then hit Ctrl+C…
d33tah
  • 10,999
  • 13
  • 68
  • 158
0
votes
0 answers

Can pyppeteer be used to interact with a Powerapps app?

We have a Powerapp that has a text box and a submission button. My goal is to control this app via pyppeteer to achieve some automation. Let me preface this by saying that I know some python, but have never used pyppeteer before, and that I know…
Block Chords
  • 1
0
votes
1 answer

How to install snapd on github codespaces?

I know that github provides ubuntu machines called codespaces. I've used one on my code which is using pyppeteer(the python version of puppeteer) to open up a browser and scrape the web but the thing is I could only get pyppeteer to work with the…
meatyari
  • 1
  • 2
0
votes
0 answers

pyppeteer pdf generation is getting killed when being downloaded

with the following code, I am using pyppeteer to generate the pdf async def generate_pdf_with_pyppeteer(self, temp_file_name, file_name): from pyppeteer import launch browser = await launch({ 'headless': True, …
Abinash Biswal
  • 63
  • 7
0
votes
0 answers

How to run multiple pyppeteer browsers for parallel processing

I need to add multiple browsers to increase the response rate for a large number of requests. At the moment the application handles 3 simultaneous requests well (for not yet cached sites) but after that the response time increases linearly. The…
mfkrg
  • 13
  • 2
0
votes
0 answers

Migrate javascript scraper to python

I have the following code in Javascript: const puppeteer = require('puppeteer'); async function run () { const browser = await puppeteer.launch(); const page = await browser.newPage(); await…
Viachaslau Pailak
  • 1
  • 1
0
votes
0 answers

How do i fix tkinter GUI freezing when i start Pyppeteer process?

I've been looking for a solution for this, but seems its more specific than most answers What im trying to do: Fix tkinter (or any other custom library) GUI freezing when the Pyppeteer process is called What i tried: Tried to put Pyppeteer…
Nefir
  • 1
  • 2
0
votes
1 answer

How to get other properties (not textContent) using Pyppeteer?

For example, in the following HTML: How do I use Pyppeteer to get the value of src? I've seen examples of this in Puppeteer, but not sure how to do it in Pyppeteer. Simply doing the following doesn't…
teles
  • 11
  • 1
0
votes
1 answer

How to click on link with specific text with Pyppeteer

I am trying to click a link with specific text. It is a button to log out from website. button = await page.xpath("//a[contains(., 'Abmelden')]") The above should select the right element via xpath. But how can I click it? Syntax is unfortunately…
weather
  • 1
  • 2
0
votes
0 answers

Pyppeteer AsyncHTMLSession never returns Arender when ran on Ubuntu + Django + Celery in async

I have a scraping function that takes a URL and uses a AsyncHTMLSession to scrape generative content off websites - the sites I'm accessing are JS heavy so this is necessary. I've also added asynchronous debugging lines to the function that log to…
Rejesto
  • 61
  • 3
0
votes
0 answers

How Pyppeteer Python connect to proxy server?

Regarding this answer: Python pyppeteer, how to use a socks proxy the proxy server for PyPpeteer in Python should be this: args = [ '--proxy-server=185.199.199.156:888', '--disable-notifications', '--start-maximized' ] `browser = await…
dev_bot_official
  • 11
  • 2
1 2 3
…
12 13