4

I've created a script using python in association with pyppeteer to keep clicking on the next page button until there is no more. The script while clicking on the next page button throws this error pyppeteer.errors.TimeoutError: Navigation Timeout Exceeded: 30000 ms exceeded. pointing at this line await page.waitForNavigation(). It can parse name and item_type from the landing page of that site, though. I know I can issue post http requests with appropriate payload to get data from there but my intention is to make use of pyppeteer and keep clicking on the next page button while parsing the required fields.

website address

import asyncio
from pyppeteer import launch

link = "https://www.e-ports.com/ships"

async def get_content():
    wb = await launch(headless=True)
    [page] = await wb.pages()
    await page.goto(link)

    while True:
        await page.waitForSelector(".common_card", {'visible':True})

        elements = await page.querySelectorAll('.common_card')
        for element in elements:
            name = await element.querySelectorEval('span.title > a','e => e.innerText')
            item_type = await element.querySelectorEval('.bottom > span','e => e.innerText')
            print(name.strip(),item_type.strip())

        try:
            await page.click("button.btn-next")
            await page.waitForNavigation()
        except Exception: break

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(get_content())

Btw, If I manually click on the next page button for the first time, It accomplishes the rest successfully.

robots.txt
  • 96
  • 2
  • 10
  • 36
  • According to the author, [this](https://github.com/miyakogi/pyppeteer/issues/58#issuecomment-377903899) is the right approach `await asyncio.wait([page.click(...), page.waitForNavigation()])`. However, it could not solve the issue either. The script starts clicking on the next page button only when I click on the button manually for the first time. – robots.txt Mar 22 '20 at 11:22

1 Answers1

1

I don't know the valid syntax in Pypeteer, but common syntax of waitForNavigation maybe this one.

await Promise.all([
   page.waitForNavigation(),
   page.click("button.btn-next")
])

With the iterator inside the array promised, all of the methods will resolved when become true or finished desired action.

Edi Imanto
  • 2,119
  • 1
  • 11
  • 17