Chrome Headless puppeteer too much CPU

Question

I have a scraping algorithm in nodejs with puppeteer which scrapes 5 pages concurrently and when it finishes with one page it pulls the next url from a queue and open it in the same page. The CPU is always at 100%. How to make puppeteer use less cpu?

This process is running on a digitaloceans droplet with 4gb of RAM and 2 vCPUs.

I've launched the puppeteer instance with some args to try to make it lighter but nothing happened

 puppeteer.launch({
    args: ['--no-sandbox', "--disable-accelerated-2d-canvas","--disable-gpu"],
    headless: true,
  });

Are there any other args I can give to make it less CPU hungry?

I've also blocked images loading

await page.setRequestInterception(true);
page.on('request', request => {
  if (request.resourceType().toUpperCase() === 'IMAGE')
    request.abort();
  else
    request.continue();
});

This is the same issue I'm having right now, also on DigitalOcean. Did you find any trick to taking care of it? — Alex MacArthur, Jul 22 '19 at 22:21

Edi Imanto · Answer 1 · 2023-07-22T16:29:03.077

31

my default args, please test it and tell me if this run smoothly. Please note that --no-sandbox isn't secure when navigate to vulnerable sites, but it's OK if you're testing your own sites or apps. So make sure, you're know what you're doing.

  const options = {
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-dev-shm-usage',
      '--disable-accelerated-2d-canvas',
      '--no-first-run',
      '--no-zygote',
      '--single-process',
      '--disable-gpu'
    ],
    headless: true
  }

  return await puppeteer.launch(options)

edited Jul 22 '23 at 16:29

answered Oct 28 '19 at 10:11

Edi Imanto

2,119
1
11
17

1

Does seem to help a LOT! Thanks! – drmrbrewer Mar 06 '20 at 21:57
6

P.S. a list of available args and their meanings is at: https://peter.sh/experiments/chromium-command-line-switches/ – drmrbrewer Mar 06 '20 at 22:34
2

Thanks for this extensive list @drmrbrewer – Edi Imanto Mar 07 '20 at 13:37
1

Thanks, but isn't it dangerous to use `--no-sandbox`? – Alex Craft Jun 10 '20 at 05:36
Anyone else have any more flags that they think will help? – hellowill89 Nov 01 '20 at 09:03
Huge -1 due to security concerns (no sandbox, really?) – The Onin Dec 15 '21 at 16:11
1

As stated that `--no-sandbox` isn't secure when you browse vulnerable sites. If you're just testing your own website, and of course you surely know what you're doing, you can use this argument. – Edi Imanto Jan 29 '22 at 17:38

score 2 · Answer 2 · answered Mar 05 '18 at 16:36

There's a few factors that can play into this. First, check if the site(s) that you're visiting using a lot of CPU. Things like canvas and other scripts can easily chew through your CPU, especially when it comes to using canvas.

If you're using docker to do your deployment then make sure you use dumb-init. There's a nice repo here that goes into why you'd use such a thing, but essentially the process ID that gets assigned in your docker image has some hiccups when it comes to handling termination:

EXPOSE 8080

ENTRYPOINT ["dumb-init", "--"]
CMD ["yarn", "start"]

This is something I've witnessed and fixed on browserless.io as I use docker to handle deployments, CPU usage being one of them.

i am using docker with dumb init, however it still chews up 50% of all 4 cores. no canvas used, only 1 page opened. — Laurynas Mališauskas, Mar 18 '18 at 09:47
@LaurynasMališauskas do you have your dockerfile someplace public? — browserless, Mar 19 '18 at 03:39

score 0 · Answer 3 · answered Jan 04 '19 at 10:54

To avoid parallel execution which causes high CPU usage , i had to execute jobs sequentially using

p-iteration NPM package. In my case it's not a problem because my jobs don't take too much time.

You can use either forEachSeries or mapSeries function depending on you scenario.

Chrome Headless puppeteer too much CPU

3 Answers3