How can I scroll a pop up element from Abnb with Selenium on Python?

Question

hey guys I'm scrapping abnb with Selenium-Python. The issue is that I want to scrap the reviews from each abnb but in order to scrap them I need to scroll down the pop up element that appears.

As you can see on the image I click the button then the pop-up element appears and then I want to scroll but I can't. Of course as you can see all the reviews don't appear and I have to scroll if I want them to appear, they also don't appear on the DevTools from google chrome unless I scroll.

I tried:

reviews_lista = []
boton_revs = driver.find_element(By.XPATH, "//button[@data-testid='pdp-show-all-reviews-button']")
boton_revs.click()

I clicked on the button that you see on the image.

I selected the whole POP UP

bloque_pop = driver.find_element(By.XPATH, "//div[@class='_14vzertx']")

I tried to scroll with this code but it didn't work

scroll = 0
while scroll < 5: # scroll 5 times
    driver.execute_script('arguments[0].scrollTop = arguments[0].scrollTop + arguments[0].offsetHeight;', bloque_pop)
    time.sleep(1)
    scroll += 1

I also tried

driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")

and this too:

bloque_pop = driver.find_element(By.XPATH, "//div[@class='_14vzertx']")
for i in range(5):
    #driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", bloque_pop)
    driver.execute_script("window.scrollTo(0,document.body.scrollHeight)", bloque_pop)
    time.sleep(1.5)

None of them work.

score 1 · Accepted Answer · answered Feb 20 '23 at 09:43

1

The easiest way to scroll inside a popup is to scrape elements contained in the popup and then scroll to the last element by using the javascript command scrollIntoView.

popup_reviews = driver.find_elements(By.CSS_SELECTOR, 'div[role=dialog] div[data-review-id]')
driver.execute_script('arguments[0].scrollIntoView({block: "center", behavior: "smooth"});', popup_reviews[-1])

answered Feb 20 '23 at 09:43

sound wave

3,191
3
11
29

So I have to execute those and that's that. I already had a similar code on the question but somehow it didn't work, after reading your answer I udnerstood and tried also with my code and it worked. Seems like after all I don't know Selenium at all. Can you tell me how it works and where did you get documented for Selenium in general? Thnk u a lot btw you saved me! – Elmauro Feb 21 '23 at 03:20
But while applying a WHILE bocule it takes too long, why would it be? – Elmauro Feb 21 '23 at 03:30
@Elmauro This is the official documentation https://selenium-python.readthedocs.io/ I suggest you start from here and test your skill with little projects – sound wave Feb 21 '23 at 07:50
@Elmauro Show me the code of the while loop so that i can understand why it is slow – sound wave Feb 21 '23 at 07:51
Nah man I'm retarded I didnt put a n+=1 in the While func and it looped forever. And for the code I had that supposedly worked... it didnt lol. Tahnks man u saved me. BTW I thought scraping the AIRBNB webpage was already a "hard" project, it isn't? – Elmauro Feb 22 '23 at 02:23
@Elmauro Well it depends by the definition of "hard" :) by the way, I suggest you starting scraping forums (for example, take a thread and scrape each message with text, username, date, likes (if any), etc) and tables. Also, I suggest you reading all my answers, some of them contain the full code to scrape something, for example: [google jobs](https://stackoverflow.com/a/75486908/8157304), [table](https://stackoverflow.com/a/75458440/8157304), [another table](https://stackoverflow.com/a/75449204/8157304), [eshop](https://stackoverflow.com/a/75447998/8157304)... – sound wave Feb 22 '23 at 07:37
1

@Elmauro ... [tweets](https://stackoverflow.com/a/75389315/8157304), [pinterest](https://stackoverflow.com/a/75386796/8157304), [bandcamp](https://stackoverflow.com/a/75334642/8157304), [another table](https://stackoverflow.com/a/75324886/8157304). Moreover, when you find a useful function or snippet code, I suggest you writing it in a file so that you can easily recall it when necessary. For example this is my simple txt file https://pastebin.com/PjgFviRC – sound wave Feb 22 '23 at 07:41

How can I scroll a pop up element from Abnb with Selenium on Python?

1 Answers1