StaleElementReferenceException while looping over list

Question

I'm trying to make a webscraper for this website. The idea is that code iterates over all institutions by selecting the institution's name (3B-Wonen at first instance), closes the pop-up screen, clicks the download button, and does it all again for all items in the list.

However, after the first loop it throws the StaleElementReferenceException when selecting the second institution in the loop. From what I read about it this implies that the elements defined in the first loop are no longer accessible. I've read multiple posts but I've no idea to overcome this particular case.

Can anybody point me in the right directon? Btw, I'm using Pythons selenium and I'm quite a beginner in programming so I'm still learning. If you could point me in a general direction that would help me a lot! The code I have is te following:

#importing and setting up parameters for geckodriver/firefox
...

# webpage
driver.get("https://opendata-dashboard.cijfersoverwonen.nl/dashboard/opendata-dashboard/beleidswaarde")

WebDriverWait(driver, 30)

# Get rid of cookie notification
# driver.find_element_by_class_name("cc-compliance").click()

# Store position of download button
element_to_select = driver.find_element_by_id("utilsmenu")
action = ActionChains(driver)
WebDriverWait(driver, 30)

# Drop down menu
driver.find_element_by_id("baseGeo").click()

# Add institutions to array
corporaties=[]
corporaties = driver.find_elements_by_xpath("//button[@role='option']")

# Iteration
for i in corporaties:
    i.click()                                                     # select institution
    driver.find_element_by_class_name("close-button").click()     # close pop-up screen
    action.move_to_element(element_to_select).perform()           # select download button
    driver.find_element_by_id("utilsmenu").click()                # click download button
    driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
    driver.find_element_by_id("baseGeo").click()                  # select drop down menu for next iteration

score 1 · Answer 1 · answered Sep 03 '21 at 10:36

This code worked for me. But I am not doing driver.find_element_by_id("utils-export-spreadsheet").click()

from selenium import webdriver
import time
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Chrome(executable_path="path")
driver.maximize_window()
driver.implicitly_wait(10)

driver.get("https://opendata-dashboard.cijfersoverwonen.nl/dashboard/opendata-dashboard/beleidswaarde")

act = ActionChains(driver)
driver.find_element_by_xpath("//a[text()='Sluiten en niet meer tonen']").click() # Close pop-up
# Get the count of options
driver.find_element_by_id("baseGeoContent").click()
cor_len = len(driver.find_elements_by_xpath("//button[contains(@class,'sel-listitem')]"))
print(cor_len)
driver.find_element_by_class_name("close-button").click()

# No need to start from 0, since 1st option is already selected. Start from downloading and then move to next items.
for i in range(1,cor_len-288): # Tried only for 5 items
    act.move_to_element(driver.find_element_by_id("utilsmenu")).click().perform()
    #Code to click on downloading option
    print("Downloaded:{}".format(driver.find_element_by_id("baseGeoContent").get_attribute("innerText")))
    driver.find_element_by_id("baseGeoContent").click()
    time.sleep(3) # Takes time to load.
    coritems = driver.find_elements_by_xpath("//button[contains(@class,'sel-listitem')]")
    coritems[i].click()
    driver.find_element_by_class_name("close-button").click()
driver.quit()

Output:

295
Downloaded:3B-Wonen
Downloaded:Acantus
Downloaded:Accolade
Downloaded:Actium
Downloaded:Almelose Woningstichting Beter Wonen
Downloaded:Alwel

score 0 · Accepted Answer · answered Sep 03 '21 at 09:56

Problem Explanation :

See the problem here is, that you have defined a list corporaties = driver.find_elements_by_xpath("//button[@role='option']") and then iterating this list, and clicking on first element, which may cause some redirection to a new page, or in a new tab etc.

so when Selenium try to interact with the second webelement from the same list, it has to come back to original page, and the moment it comes back, all the elements become stale in nature.

Solution :

one of the basic solution in this cases are to define the list again, so that element won't be stale. Please see the illustration below :-

Code :

corporaties=[]
corporaties = driver.find_elements_by_xpath("//button[@role='option']")
# Iteration
j = 0
for i in range(len(corporaties)):
    elements = driver.find_elements_by_xpath("//button[@role='option']")
    elements[j].click()
    j = j + 1 # select institution
    driver.find_element_by_class_name("close-button").click()     # close pop-up screen
    action.move_to_element(element_to_select).perform()           # select download button
    driver.find_element_by_id("utilsmenu").click()                # click download button
    driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
    driver.find_element_by_id("baseGeo").click()  # select drop down menu for next iteration
    time.sleep(2)

Thank you for your clear explanation! Should have come up with the idea to determine the length of the list outside the loop and getting the element inside it. I do have a follow up question though. Now it throws one of two errors: either that the `Element — Sukaldi, Sep 03 '21 at 11:19
I set the `time.sleep(2)` to five seconds. That seemed to do the trick! Your help is much appreciated! — Sukaldi, Sep 03 '21 at 11:29

StaleElementReferenceException while looping over list

2 Answers2

Problem Explanation :

Solution :

Linked