pyppeteer: page.click and page.waitForNavigation issues while going through multiple pages

Question

I'm learning puppeteer in JavaScript and following a book and some documentation and tutorials found online. I found a good tutorial going through multiple pages of a famous online shop and saving the items in a file. The JavaScript code I wrote following this tutorial, changing what had to be changed, is working well. The problem is with my Python porting using pyppeteer

I had the issue described here https://github.com/miyakogi/pyppeteer/issues/58 and applied the solution in the following code

import asyncio, json
from pyppeteer import launch

async def main():   
    browser = await launch(headless = False, defaultViewport = False)
    page = await browser.newPage()
    await page.goto(
        "https://shop_site_link",
    {
        'waitUntil': "load"
    })

    items = []
    item_keys = ['title','price','img']

    isBtnDisabled = False

    while (not isBtnDisabled):
        await page.waitForSelector('[data-cel-widget="search_result_0"]')
        ProductHandles = await page.querySelectorAll( 
            "div.s-main-slot.s-result-list.s-search-results.sg-row > .s-result-item"
        )#this replace page.$$( "div.s-main-slot.s-result-list.s-search-results.sg-row > .s-result-item");


        for producthandle in ProductHandles:
            title = None
            price = None
            img  =  None
            
            try:
                title = await page.evaluate('''
                el => el.querySelector("h2 > a > span").textContent
                ''', producthandle)
            except:
                print('some error')

            try:
                price = await page.evaluate('''
                el => el.querySelector(".a-price > .a-offscreen").textContent
                ''', producthandle)
            except:
                print('some error')  

            try:
                img = await page.evaluate('''
                el => el.querySelector(".s-image").getAttribute("src")
                ''', producthandle)
            except:
                print('some error')             
            
            if (title is not None):
                items.append(dict(zip(item_keys, [title, price, img])))

        is_disabled =  await page.querySelector('.s-pagination-item.s-pagination-next.s-pagination-disabled')!=None
        isBtnDisabled = is_disabled;
        if (not is_disabled):
            await asyncio.wait([
                page.click(".s-pagination-next"),
                page.waitForSelector(".s-pagination-next", { 'visible': True }),
                page.waitForNavigation({'waitUntil' : "networkidle2"},timeout=15000)
            ])

    #await browser.close()
    print(len(items))

    with open('items.json', 'w') as f:
        json.dump(items, f, indent = 2)

    # with open('items.json', 'r') as readfile:
    #     print(json.load(readfile))
asyncio.get_event_loop().run_until_complete(main())

as per issue described in pyppeteer github I issued the page.click and page.waitForNavigation at "same time" this way

        if (not is_disabled):
        await asyncio.wait([
            page.click(".s-pagination-next"),
            page.waitForSelector(".s-pagination-next", { 'visible': True }),
            page.waitForNavigation({'waitUntil' : "networkidle2"},timeout=15000)
        ])

trying to do what I do in the JavaScript code here:

if (!is_disabled) {
    await Promise.all([
        page.click(".s-pagination-next"),
        page.waitForNavigation({ waitUntil: "networkidle2" }),
    ]);

}

Now, the issue and related question is, the code works well but I receive the following warning:

DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.

anyone knows a better implementation that will work well with Python 3.11?

score 1 · Answer 1 · answered Jan 08 '23 at 17:30

at the end I found a solution that maybe useful for whomever will have the same issue

        if (not is_disabled):
            if sys.version_info < (3, 11):
                await asyncio.wait([
                    page.click(".s-pagination-next"),
                    page.waitForSelector(".s-pagination-next", { 'visible': True }),
                    page.waitForNavigation({'waitUntil' : "networkidle2"},timeout=15000)
                    ])
            else:
                async with  asyncio.TaskGroup() as tg:
                    task1 = tg.create_task(page.click(".s-pagination-next"))
                    task2 = tg.create_task(page.waitForNavigation({'waitUntil' : "networkidle2"},timeout=15000))

so basically to use TaskGroup in order to wait to all tasks to complete

score 0 · Answer 2 · answered Apr 26 '23 at 15:40

The documentation of pyppeteer click recommends using asyncio.gather.

So, for your code, it would be something like this:

await page.waitForSelector(".s-pagination-next", { 'visible': True })

await asyncio.gather(
    page.click(".s-pagination-next"),
    page.waitForNavigation({ 'waitUntil': "networkidle2" }, timeout=15000)
)

That does not give the DeprecationWarning for Python versions 3.8 - 3.11.

Note that gather does not necessarily run each coroutine sequentially, so waiting for the selector should be awaited first.

pyppeteer: page.click and page.waitForNavigation issues while going through multiple pages

2 Answers2