I want to use pyppeteer to open a webpage, wait for the search box to appear, fill in a search query, submit and wait for the new page's content (obtained after clicking the submit button) to appear.
I have tried using the following code with page.evaluate and javascript to simulate the above tasks.
Javascript
script = """
() => {
async function waitForElm(selector) {
return new Promise(resolve => {
if (document.querySelector(selector)) {
return resolve(document.querySelector(selector));
}
const observer = new MutationObserver(mutations => {
if (document.querySelector(selector)) {
resolve(document.querySelector(selector));
observer.disconnect();
}
});
observer.observe(document.body, {
childList: true,
subtree: true
});
});
}
waitForElm('input').then( function() {
document.getElementsByClassName("px-3 py-2 outline-none rounded-none disabled:bg-gray-200")[0].setAttribute("value", "%s");
document.getElementsByClassName("px-3 py-2 outline-none rounded-none disabled:bg-gray-200")[0].click();
document.getElementsByClassName("px-3 py-2 outline-none rounded-none disabled:bg-gray-200")[0].dispatchEvent(new Event('change', { 'bubbles': true }));
document.getElementsByClassName('bg-secondary-600 text-white py-1 font-bold shadow-lg')[0].click();
}).then(function() { return waitForElm('button[class="px-6 py-2 bg-primary-500 hover:bg-primary-700 text-white font-medium rounded"]');});
}
"""%(query)
Code not giving correct content
url='https://pricehistoryapp.com/'
page = await self.browser.newPage()
await page.goto(url, options={'timeout': 125000})
result = await page.evaluate(script)
content = await page. Content()
But I am still getting the output 'content' as the page which should be before doing the search query.
I have also tried using waitUntil: 'networkidle0', Page.waitForNetworkIdle(), waitForSelector methods but still no help.
But I get the correct content of page (i.e. after button click) if I use sleep i.e. :
Code giving correct content but constant sleep
page = await self.browser.newPage()
await page.goto(url, options={'timeout': 125000})
result = await page.evaluate(script)
await asyncio.sleep(29)
content = await page. Content()
I don't know what mistake I am making please help.