Stack Overflow
Stack Overflow

Questions tagged [puppeteer-cluster]

puppeteer-cluster manages a pool of headless browsers via puppeteer. This is useful to crawl multiple pages in parallel or to keep a pool of open browsers.

puppeteer-cluster creates a pool of puppeteer workers by spawning multiple browsers, contexts or pages via puppeteer. The library keeps track of queued jobs and handles thrown errors. In addition, it allows to retry jobs or introduce delays when crawling a domain.

Resources:

73 questions
0
votes
1 answer

Errors with page.$$eval() using Puppeteer

I'm trying to use page.$$eval() to get data from an
    , store the values in two arrays and then iterate through the arrays to put the data into key/value pairs. I'm struggling with getting the data into the arrays. The HTML looks like this:
    user7478189
0
votes
0 answers

How can I work with Jest and puppeteer-cluster

I am currently doing 40+ tests (they can take up to 20+ minutes and will grow unfortunately) for a web-app, and was thinking of running multiple browsers in order to run tests in parallel. I am currently trying to work with puppeter-cluster but had…
ka_lin
  • 9,329
  • 6
  • 35
  • 56
0
votes
1 answer

How to store data from multi page to json?

thank you for ur attention, so i write a mini project that scrape news site and store main texts from them. i tried many solutions to add json in my project without have consol.log but always after scraping its show only one main text. so i show my…
Nersi ArchiMage
  • 5
  • 2
0
votes
1 answer

Issue with puppeteer cluster scripts in AWS EC2 Instance

I'm facing issue in page loading when running through puppeteer(puppeteer-cluster) scripts in AWS EC2 Instance. But the same is working when I m opening manually in EC2. Also, puppeteer scripts working perfectly in local machines both Headless…
Devaraj
  • 1
  • 1
0
votes
1 answer

How to use puppeteer.connect() in puppeteer-cluster?

I want to use a remote headless browser service (this one) and in the docs they use puppeteer.connect() instead of puppeteer.launch() this the code to launch the puppeteer cluster: const cluster = await Cluster.launch({ concurrency:…
Anas Latique
  • 357
  • 5
  • 13
0
votes
1 answer

pupeteer not able to navigate to a new URL after clicking on button on jenkins

After clicking on a button puppeteer not navigating to new URL. console.log(this.page.url);//current URL await this.utils.click(this.page, '[trackid=navigation_primary_button] :not([disabled])' ); console.log(this.page.url);//current URL // expected…
user10478566
  • 37
  • 12
0
votes
0 answers

How to reuse event added page in cluster.task? (do not close page after task, and want to reuse it)

In cluster.task, I added some event like framenavigated with complex logic. I use puppeteer-cluster in my API server, when request get in, adding in the queue, and workers do the job sequencely. But every time request comes, framenavigated event…
Yangha Koo
  • 17
  • 3
0
votes
1 answer

Puppeteer performance leaks

I am using a puppeteer and a puppeteer-cluster to take screenshots of the received HTML. To "bypass" the Node.js' single-thread processing, I use several Docker containers. The problem is that it randomly prolongs a render time - once the same HTML…
KRiSTiN
  • 429
  • 1
  • 4
  • 11
0
votes
1 answer

Puppeteer-cluster using tab and taking screenshot

I am using puppeteer-clustor and imagemagick (convert) / xwd command to take screenshot of complete desktop. Would need browser with viewable part of page and browser navigation buttons and URL. I could get screenshot most of the times, however it…
Chakra
  • 66
  • 1
  • 7
0
votes
1 answer

Running Puppeteer test multiple times with different scenarios Async/Await

Hi Guys I need some help for a scenario I have an Array with say filled with Google, Yahoo 100 objects. If Google do X Else Yahoo do Y This is easy in say Java Selenium, just loop through with an if statement, and start stop browser, given…
Elsid
  • 243
  • 2
  • 10
  • 25
-1
votes
1 answer

Puppeteer-cluster close browsers itself

I use puppeteer-cluster + node js. I have some script. I'm a newbie The script runs halfway through and then ends and is not executed further. I've been looking for a solution for a week now, I don't understand what's the matter. Help me please…
nobdefender
  • 13
  • 2
-4
votes
2 answers

I want to create Infinite loop in puppeteer-cluster

I want create infinite loop in puppeteer-cluster My Code but After about 30 seconds, it caused idle What should I do?
Raksit kongwong
  • 1
  • 5
-6
votes
1 answer

element is not visible on site using puppeteer

A chat start button exist in site as shown below but while running automated script on puppeteer this button doesn't display. Even i used all kinds waits for it to load. as here below button doesn't appear
Muneeb Khan
  • 19
  • 6
1 2 3 4
5