Puppeteer issue: Same button cannot be pressed twice after page content changes (but button id did not)

Question

I'm trying to automate an checkout process using latest nodeJS and puppeteer. Everything is working except the last button press, which should be working from my understanding, but regardless nothing is happening.

Here is my code:

            else if (statusCode == 200) {
            let data = await page.evaluate(() => document.body.innerHTML);
            // Accepting cookies
            const cookieButton = 'button[id="privacy-layer-accept-all-button"]';
            await page.waitForSelector(cookieButton);
            await page.click(cookieButton);
            // Confirm cart data
            bench = new Date().getTime();
            let checkoutButton = 'div[data-test="checkout-continue-desktop-enabled"]';
            await page.waitForSelector(checkoutButton);
            await page.click(checkoutButton);
            await page.waitForNavigation({waitUntil:'networkidle0'})
            // Confirming checkout
            data = await page.evaluate(() => document.body.innerHTML);
            checkoutButton = 'div[data-test="checkout-continue-desktop-enabled"]';
            await page.waitForSelector(checkoutButton);
            await page.click(checkoutButton);

Code of the first Button (confirming cart data) - which is getting clicked (works):

<div><div class="BasketResumeHeader__BasketResumeHeaderWrapper-k9sckg-0 dWBNeC"><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Zusammenfassung</h3><span font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 ejSGsu">Gutscheine können im Zahlungsschritt hinzugefügt werden</span></div><div class="ResumeTable__StyledTableValues-sc-109drs2-0 hUoXfQ"><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 dHuLzW"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div>Zwischensumme</div><span>101,99&nbsp;€</span></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 eYsMjW">Lieferkosten<div color="#918e8c" height="20" width="20" class="Icon-sc-1vrq823-0 cScGKb InfoIcon__StyledInfoIcon-sc-1vh1773-0 clmMtr"><svg width="32" height="32" viewBox="0 0 24 24"><path d="M12 2a10 10 0 1010 10A10 10 0 0012 2zm0 18a8 8 0 118-8 8 8 0 01-8 8zm1-7.5a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v3a.5.5 0 00.5.5h1a.5.5 0 00.5-.5zm0-4a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v1a.5.5 0 00.5.5h1a.5.5 0 00.5-.5z"></path></svg></div></div><div>Gratis</div></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Gesamtsumme</h3><div class="mms-checkout-basket-summary__vat"><span color="grey2" font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 blKxsn">inkl. MwSt.</span></div></div><h3 color="black" font-family="default" spacing="base" font-size="lg" data-test="checkout-total" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN"><span>101,99&nbsp;€</span></h3></div><div class="BasketResume__StyledSummaryButtons-sc-1pu65vy-0 fFBBbE"><div data-test="checkout-continue-desktop-enabled" class="ContinueButton__StyledContinue-sc-17lzxkg-0"><button type="button" class="Buttonstyled__StyledButton-sc-140xkaw-1 jjXwnX">Zur Kasse gehen</button></div><a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 cSeNct BasketResume__StyledLink-sc-1pu65vy-1 duDsob" target="_self" href="/" color="black"><span color="black" font-size="sm" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 iMnkvI Linkstyled__Text-sc-1drhx1h-4 Uqzhg">Weiter einkaufen</span></a></div></div>

Code of the second button to confirm the checkout page (whole URL changes from /checkout to checkout/summary in the process) - does not work and doesn't get clicked sadly:

<div><div class="BasketResumeHeader__BasketResumeHeaderWrapper-k9sckg-0 dWBNeC"><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Zusammenfassung</h3><a class="Linkstyled__StyledLinkRouter-sc-1drhx1h-2 kgubGI BasketResumeHeader__StyledLink-k9sckg-1 dEwdDh" data-test="mms-router-link" href="/checkout/payment#discounts"><span class="Linkstyled__StyledIconWrapper-sc-1drhx1h-3 ctFeV"><div color="primary" class="Icon-sc-1vrq823-0 cRMyVe"><svg width="32" height="32" viewBox="0 0 24 24"><path d="M15.78 11.28a.75.75 0 01.22.53v.38a.77.77 0 01-.22.53l-5.14 5.13a.5.5 0 01-.71 0l-.71-.71a.49.49 0 010-.7L13.67 12 9.22 7.56a.5.5 0 010-.71l.71-.7a.5.5 0 01.71 0z"></path></svg></div></span><span color="grey5" font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 jjFdpe Linkstyled__Text-sc-1drhx1h-4 Uqzhg">Haben Sie einen Gutscheincode oder eine Geschenkkarte?</span></a></div><div class="ResumeTable__StyledTableValues-sc-109drs2-0 hUoXfQ"><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 dHuLzW"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div>Zwischensumme</div><span>101,99&nbsp;€</span></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 eYsMjW">Lieferkosten<div color="#918e8c" height="20" width="20" class="Icon-sc-1vrq823-0 cScGKb InfoIcon__StyledInfoIcon-sc-1vh1773-0 clmMtr"><svg width="32" height="32" viewBox="0 0 24 24"><path d="M12 2a10 10 0 1010 10A10 10 0 0012 2zm0 18a8 8 0 118-8 8 8 0 01-8 1zm1-7.5a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v3a.5.5 0 00.5.5h1a.5.5 0 00.5-.5zm0-4a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v1a.5.5 0 00.5.5h1a.5.5 0 00.5-.5z"></path></svg></div></div><div>Gratis</div></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Gesamtsumme</h3><div class="mms-checkout-basket-summary__vat"><span color="grey2" font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 blKxsn">inkl. MwSt.</span></div></div><h3 color="black" font-family="default" spacing="base" font-size="lg" data-test="checkout-total" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN"><span>101,99&nbsp;€</span></h3></div><div class="BasketResume__StyledSummaryButtons-sc-1pu65vy-0 fFBBbE"><div data-test="checkout-continue-desktop-enabled" class="ContinueButton__StyledContinue-sc-17lzxkg-0"><button type="button" class="Buttonstyled__StyledButton-sc-140xkaw-1 jjXwnX">Fortfahren und bezahlen</button><p font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 iEQpGm ContinueButton__StyledDescription-sc-17lzxkg-1 dGihln">Mit Klick auf "Fortfahren und bezahlen" stimme ich den <a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 RsClv Summarystyled__StyledLink-sc-1euxzmh-10 fYjVAT" target="_blank" rel="noopener noreferrer" href="https://www.mediamarkt.de/de/legal/terms">AGB</a> zu. Ich habe die <a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 RsClv TransLink__StyledLink-sc-1f101w2-0 gwpXWL" target="_blank" rel="noopener noreferrer" href="https://www.mediamarkt.de/de/legal/terms#terms.withdrawal">Widerrufsbelehrung</a> und die <a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 RsClv TransLink__StyledLink-sc-1f101w2-0 gwpXWL" target="_blank" rel="noopener noreferrer" href="https://www.mediamarkt.de/de/shop/datenschutzhinweis_shop.html">Datenschutzerklärung</a> zur Kenntnis genommen.</p></div></div></div>

Sadly I can't give you better examples/hyperlinks other than those code snippets as the sites won't load without a working cookie on the site.

As you can see the selector should select the button (as it has exactly the same name on both pages), but somehow it struggles to press the button after the page gets changed after the first button was pressed successfully. I tried to add await page.waitForNavigation();, { waitUntil: 'networkidle0'} or await page.waitForTimeout(1000); because I figured maybe the data did not load in yet, but it did not really help and I do not seem to understand the problem why it isn't working.

The error message I do get is:

(node:16524) UnhandledPromiseRejectionWarning: Error: Node is either not visible or not an HTMLElement

Would appreciate if you could help me out with this problem or at least pinpoint me in the right direction how to further debug and solve this problem.

score 1 · Answer 1 · answered May 11 '21 at 15:32

I have a few suggestions. Add this in to ensure that your viewing window is a good size:

await page.setViewport({width: 1024, height: 1600});

You can try clicking the button directly within the web browser context using evaluate:

await page.evaluate(selector=>{
  document.querySelector(selector).click();
},checkoutButton);

You may need to scroll to the element:

await page.evaluate(selector=>{
  elem = document.querySelector(selector);
  elem.scrollIntoView({ block: "center", inline: "center" });
  elem.click();
},checkoutButton);

As a debug/test tool, take a screenshot to see if the page after loading looks like what you expect:

await page.screenshot({path: 'test.png'});

Thanks a lot for your reply! I am already using `await page.setViewport({ width: 1920, height: 1080 })` and `headless: false, defaultViewport: null, args: ['--start-maximized']`. So I see that the page loads properly, the button is there (and can easily be clicked manually), when I use cheerio to get its text it works too (prints "Fortfahren und bezahlen" to console when I do `console.log($('button[id="privacy-layer-accept-all-button"]').text);`)... but somehow I cannot select it or click it anymore via Puppeteer. Related: https://github.com/puppeteer/puppeteer/issues/1771 — l4m0r, May 12 '21 at 03:08

score 1 · Answer 2 · answered May 11 '21 at 16:35

1

From the puppeteer documentation, they recommend that you combine the click and waitForNavigation promises into a single promise and then wait for them both to resolve, this is because if you did each one of them separately, it may get stuck because waitForNavigation may get finished before click is resolved, for this reason, change this:

await page.click(checkoutButton);
await page.waitForNavigation({waitUntil:'networkidle0'})

To this:

await Promise.all([
page.click(checkoutButton),
page.waitForNavigation({waitUntil:'networkidle0'})
])

answered May 11 '21 at 16:35

Movianlost

226
2
8

Thanks for your reply but it did not work and the page being stuck is not the problem here. The problem seems to be that it cannot find the button at all after firing the button for the first time, even when I use `await page.waitForTimeout(10000);` and manually watch the page finish to load (and can inspect the button via dev tools). This is almost 100% identical to my problem, but sadly this method did not work for me as well :( https://github.com/puppeteer/puppeteer/issues/1771 – l4m0r May 12 '21 at 03:06
@l4m0r is there anything overshadowing the button? the click method on puppeteer is different from the built-in click method. The puppeteer click fires mouse clicks, so if the button is not visible on the page or there is another element on top of it, it won't work. Aside from this, I can't think of anything that could cause this problem without seeing the website you are scraping. – Movianlost May 12 '21 at 03:43
Thanks for your reply again. No, there is nothing overshadowing the button. Also I do not click the button with cheerio (does that even work?), I just print its text to see if cheerio can find it (or the previous button with different text)... So cheerio can access the updated button, but puppeteer can't. The website is: https://www.mediamarkt.de/checkout (1st button) and https://www.mediamarkt.de/checkout/summary (2nd button). For the 2nd url/button you are going to need an account, though – l4m0r May 12 '21 at 03:49

Puppeteer issue: Same button cannot be pressed twice after page content changes (but button id did not)

2 Answers2