0

So I created my first web scrapper with Puppeteer and it works fine. I call the scrapper function in my server at 5mins time intervals but after calling it a couple times about 10 to 15 times, i begin to get this timeout error TimeoutError: waiting for selector `body > div.page-window.market-watch.compact > div > div.b > div.page-block > div > table > tbody > tr:nth-child(1) > td.symbol` failed: timeout 30000ms exceeded and I am sure the selector path is correct, the function even works well until after about 10 calls, then it breaks and would only work again if I restart the server. Please how can i fix this problem? My code bellow

puppeteer
    .launch({
      headless: true,
      args: ["--no-sandbox", "--disable-setuid-sandbox"],
    })
    .then(async (browser) => {
      console.log("Puppeteer Has launched...");
      const page = await browser.newPage();
      await page.setDefaultNavigationTimeout(120000); // set default timeout to 2mins
      try {
        await page.goto(baseUrl, { waitUntil: "load", timeout: 120000 });
        // wait for price section to mount
        await page.waitForSelector(
          "body > div.page-window.market-watch.compact > div > div.b > div.page-block > div > table > tbody > tr:nth-child(1) > td.symbol"
        );
      } catch {
         console.log(error)
      }

also as you can see in my code, i set the default timeout to be on 2mins but i get a timeout error of 30secs. Can anyone please explain why too?

Ebuka
  • 19
  • 1
  • 9
  • What's the page? Could be pretty much anything that might cause this, but in general I [caution against these ultra-specific, brittle selectors](https://serpapi.com/blog/puppeteer-antipatterns/#misusing-developer-tools-generated-selectors). – ggorlen Jun 23 '22 at 23:25
  • https://trade.mql5.com/trade?servers= this is the page ggorlen. Thanks so much. But you might need to login on a demo account. Would that be a problem for you? – Ebuka Jun 24 '22 at 01:24
  • and ggorlen, please do you have any idea on why my page.setDefaultNavigationTimeout(120000) isn't setting the default timeout to 2mins. As i go a 30secs timeout error – Ebuka Jun 24 '22 at 01:38
  • You probably want [`setDefaultTimeout`](https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#pagesetdefaulttimeouttimeout) or set the timeout directly on the `waitForSelector` call. But that probably won't help you--30 sec is usually plenty of time, so there's probably a deeper issue. I'll take a look at the actual problem if I get the chance, but I suspect the site is detecting your bot and throttling access, there's an A/B test where selectors change, a pop-up that only shows up every so often, or something like that. I'd run headfully or print the HTML and see if you can tell. – ggorlen Jun 24 '22 at 01:47
  • Alright. Thanks so much ggorlen – Ebuka Jun 24 '22 at 04:51
  • Hi @ggorlen it seems what happens is that, after a scrapping for a little time, the site detects I am a bot and then changes the selector path. This same issue has happened to another section in my code. Do you have any posible or suggested solution? I am also using puppeteer-extra-stealth-plugin as well. But i still get detected in the end – Ebuka Jun 24 '22 at 08:21

0 Answers0