Async puppeteer browser disconnect after few iterations

Question

I tested iterations with puppeteer in a small case. I already have read the common reason for puppeteer disconnections are that the Node script doesnt wait for the puppeteer actions to be ended. So I converted all functions in my snippet into async functions but it didnt help.

If the small case with six iterations work I will implement it in my current project with like 50 iterations.

'use strict';

const puppeteer = require('puppeteer');

const arrIDs = [8322072, 1016816, 9312604, 1727088, 9312599, 8477729];

const call = async () => {
    await puppeteer.launch().then(async (browser) => {
        arrIDs.forEach(async (id, index, arr) => {
            await browser.newPage().then(async (page) => {
                await page.goto(`http://somelink.com/${id}`).then(async () => {
                    await page.$eval('div.info > table > tbody', async (heading) => {
                        return heading.innerText;
                    }).then(async (result) => {
                        await browser.close();
                        console.log(result);
                    });
                });
            });
        });
    });
};

call();

score 2 · Accepted Answer · answered Feb 11 '20 at 23:14

forEach executes synchronously. replace forEach with a simple for loop.

const arrIDs = [8322072, 1016816, 9312604, 1727088, 9312599, 8477729];
const page = await browser.newPage();

for (let id of arrIDs){
    await page.goto(`http://somelink.com/${id}`);
    let result = await page.$eval('div.info > table > tbody', heading => heading.innerText).catch(e => void e);
    console.log(result);
}

await browser.close()

score 1 · Answer 2 · answered Feb 11 '20 at 20:44

The way you've formatted and nested everything seems like some incarnation of callback hell.

Here's my suggestion, its not working, but the structure is going to work better for Async / Await

const puppeteer = require("puppeteer");

const chromium_path_706915 =
  "706915/chrome.exe";

async function Run() {

    arrIDs.forEach(
        await Navigate();
    )
  async function Navigate(url) {
    const browser = await puppeteer.launch({
      executablePath: chromium_path_706915,
      args: ["--auto-open-devtools-for-tabs"],
      headless: false
    });

    const page = await browser.newPage();

    const response = await page.goto(url);

    const result = await page.$$eval("div.info > table > tbody", result =>
      result.map(ele2 => ({
        etd: ele2.innerText.trim()
      }))
    );


    await browser.close();
    console.log(result);
  }
}

run();

score 1 · Answer 3 · answered Feb 11 '20 at 23:23

On top of the other answers, I want to point out that async and forEach loops don't exactly play as expected. One possible solution is having a custom implementation that supports this:

Utility function:

async function asyncForEach(array: Array<any>, callback: any) {
  for (let index = 0; index < array.length; index++) {
    await callback(array[index], index, array);
  }
}

Example usage:

const start = async () => {
  await asyncForEach([1, 2, 3], async (num) => {
    await waitFor(50);
    console.log(num);
  });
  console.log('Done');
}

start();

Going through this article by Sebastien Chopin can help make it a bit more clear as to why async/await and forEach act unexpectedly. Here it is as a gist.

Thank you a lot about the `forEach` behaviour information and the article. — phng, Feb 12 '20 at 09:26

Async puppeteer browser disconnect after few iterations

3 Answers3