0

I am trying to use page.evaluate in Pyppeteer and capture js script response but I am unable to capture it. In the following code, I am trying to capture the result returned by js script in dimensions variable, but its capturing as None

import asyncio
from pyppeteer import launch
async def hmm():
    browser = await launch(headless=False)
    page = await browser.newPage()
    await page.goto('https://jobs.chegg.com')
    dimensions = await page.evaluate("""async () => {
        var script = document.createElement('script');
        script.type = 'text/javascript';
        script.src = 'https://cdnjs.cloudflare.com/ajax/libs/axe-core/3.1.2/axe.min.js';
        document.head.appendChild(script);
        var result = new Promise((resolve, reject) => {
            axe.run(document, {
            runOnly: {
                type: "tag",
                values: ["wcag2a", "wcag2aa", "best-practice"]
            },
            "rules": {
                "skip-link": { enabled: false }
            }
            }, function(err, results) {
            if (err) throw err;
                resolve(results);
            });
        });
        let test = await result.then((res) => {
            return res;
        }).catch(err => {
            console.log(err);
        });

        console.log(test); 
        return test;
        }

    """)
    print(dimensions) # None
    return dimensions
asyncio.get_event_loop().run_until_complete(hmm())

Note :- Open console in any website and run the js script, then, an object/dictionary is returned.

Please suggest a workaround for this problem.

Mahesh
  • 1,117
  • 2
  • 23
  • 42

1 Answers1

2

Not sure what you are trying to accomplish but if you want to capture response from the site, you should listen to response. Here is an example with your site. It will print out every response object and related information.

import asyncio
from pyppeteer import launch

def interception_fun(response):
    # Response logic goes here
    print("URL:", response.url)
    print("Method:", response.request.method)
    print("Response headers:", response.headers)
    print("Request Headers:", response.request.headers)
    print("Response status:", response.status)
    return


async def hmm():
    browser = await launch(headless=False)
    page = await browser.newPage()
    await page.goto('https://jobs.chegg.com')
    page.on('response', interception_fun)
    
    await browser.close()

    return
asyncio.get_event_loop().run_until_complete(hmm())

Update: As of pyppeteer version 0.2.5. page.on() should be lambda function like this:

page.on('response', lambda res: asyncio.ensure_future(interception_fun(res)))

Ernest
  • 186
  • 10
  • this doesn't work when u want response.json() – Abhishek Kumar Jun 06 '21 at 13:33
  • @AbhishekKumar It works! You have to make sure that response is a json. Like this - `if "application/json" in response.headers.get("content-type", ""):` and then await response json like `r = await response.json()`. Now you have your json. – Ernest Jun 29 '21 at 11:55
  • yes figured that `await` part later, now it works. thanks – Abhishek Kumar Jun 30 '21 at 14:42
  • pls [check](https://stackoverflow.com/questions/69879614/python-pyppeteer-intercept-capture-network-requests) – Pyd Nov 08 '21 at 06:58