1

I need to visit some independent URLs where the elements of the pages are loaded very slowly. It usually takes several minutes to make the entire page completely loaded. However, only a small portion of this page is useful. The useful part can be indicated by a certain selector on the page. Therefore, I would like to know whether I can tell puppeteer to stop wait for the page once the key selector has already appeared, to accelerator the speed of the . There are extensive answers tell us to use await page.waitForSelector('.class_sample');

So I use it like this:

page = await browser.newgpae();
await page.goto('example.com/xxx.html');
await page.waitForSelector('.class_sample');`

However, it still stucks on the stage of page.goto(). Sometimes pyppeteer reports a timeout error after 30s as the targeted page is too slow.

I've found that most of the examples about the method waitForSelector() are put behind a .click() method. My troublesome case is that the pages are independent of each other (example.com/xxxxx.html) and cannot be visit by a click on a link, so a waitForSelector() method doesn't solve my problem yet.

Any suggestion would be greatly appreciated.

Bo Wang
  • 55
  • 7
  • You have some typos and a missing `await` in your first line: `browser.newgpae();` should be `await browser.newPage();` . Was it a mistake in you question, or these errors present in your real script as well? – theDavidBarton Jun 19 '20 at 14:36
  • 1
    Sorry. It's just a careless typing mistake in my question. I revised my question, trying to make it more understandable. Look forward to your further help and thank you in advance. – Bo Wang Jun 22 '20 at 09:28
  • I see. See my solution with `{ waitUntil: 'domcontentloaded' }` if it helps to make the script more stable. – theDavidBarton Jun 22 '20 at 09:40
  • Actually my requirement is accelerating the navigation by stoping it in advance if puppeteer find that the special element has appeared. The 'domcontentloaded' event will make chrome wait until the entire page is loaded including some slow javascripts. So is it possible to use waitForSelector event in goto() method? – Bo Wang Jun 22 '20 at 09:54
  • No, `waitForSelector` is not an option in [page.goto](https://pptr.dev/#?product=Puppeteer&version=v4.0.0&show=api-pagegotourl-options). Do you use any `page.setRequestInterception`? Maybe you can speed up loading with skipping certain request types. – theDavidBarton Jun 22 '20 at 11:45

1 Answers1

0

I suggest to use page.goto with the domcontentloaded option. Why? Because the default option is the load and it waits for much longer time for those elements you said are not required for you, while your selector will be most probably already available when the DOMContentLoaded event fires.

I suggest to leave the waitForSelector as well, it will makes your script more reliable.

await page.goto('example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.class_sample');

DOMContentLoaded

The DOMContentLoaded event fires when the initial HTML document has been completely loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading. [source]

load

The load event is fired when the whole page has loaded, including all dependent resources such as stylesheets and images. This is in contrast to DOMContentLoaded, which is fired as soon as the page DOM has been loaded, without waiting for resources to finish loading. [source]

theDavidBarton
  • 7,643
  • 4
  • 24
  • 51