Clicking on "load more" button via puppeteer

Question

I am new to JS. I need to parse comments from Instagram, but first I need to load them all. I am using Puppeteer in Node.js, so I wrote this code:

await page.evaluate(() => {
    while(document.querySelector('main').querySelector('ul').querySelector('button'))
      document.querySelector('main').querySelector('ul').querySelector('button').click()
  })

It does nothing and starts an endless loop. I tried to make a timeout inside the loop and so on... I expect that code will check if this button exists and if true - click() on it while it exists loading more and more comments. I can't catch what am I doing wrong.

score 1 · Accepted Answer · answered Nov 19 '18 at 15:19

1

Have a look at my answer to a question very similar to this one here:
Puppeteer / Node.js to click a button as long as it exists -- and when it no longer exists, commence action

You should be able to apply it to finding and continually clicking on your "load more" button.

answered Nov 19 '18 at 15:19

AJC24

3,280
2
19
30

Thanks a lot! Am I right writing like that because I get 'undefind' `const loadMoreButton = await page.evaluate(() => { return document.querySelector('main').querySelector('ul').querySelector('button'); })` ? – ExsaNik Nov 19 '18 at 15:50
I would suggest you use a cleaner CSS selector. For example you don't need to query for `main`, then for `ul`, then for `button`. Depending on the layout of your DOM, you might be able to do something simpler such as `ul li button` (assuming the button is inside a list element in your unordered list). Keep everything clean and simple. Don't over-complicate it. If you want, post a snapshot of your DOM and which button you're trying to find and then we can all help you find a simpler selector. – AJC24 Nov 19 '18 at 15:56
I tried with `const loadMoreButton = '.class'` and it works perfectly, thanks! But I need to access though dom elements because instagram often changes its classes – ExsaNik Nov 19 '18 at 15:59
If classes are changing (and I'm aware a number of sites do this) then what you need to do is find a unique part of the selector that never changes. For example, on that button the class might be `2xP123_button` today but `3rQ456_button` next month: note that the `_button` part of the class stays the same all of the time. So you can have a selector like this: `button[class*="_button"]` to indicate that you want to find that button. – AJC24 Nov 19 '18 at 16:03

score 0 · Answer 2 · answered Nov 19 '18 at 16:16

Instead of a using a while() loop, you can use setInterval() to slow down each iteration to a more manageable pace while you load the comments:

await page.evaluate(async () => {
  await new Promise((resolve, reject) => {
    const interval = setInterval(() => {
      const button = document.querySelector('main ul button');
      if (button !== null) {
        button.click();
      } else {
        clearInterval(interval);
        resolve();
      }
    }, 100);
  });
});

Clicking on "load more" button via puppeteer

2 Answers2