Selenium 4 (python): stale element reference during table web scraping

Question

I am web scraping a web table that looks like the follow:

 |   A  |    B   |   C   |      D                    |
1| Name | Surname| Route | href="link with more info"| 
2| Name | Surname| Route | href="link with more info"| 
3| Name | Surname| Route | href="link with more info"|

links = driver.find_elements(by='xpath', value='//a[@title="route detail"]')

So far so good, I get what I want.

Now I want to click on the collected links to gather the info in the subpage (which I know how) and return to the main page, move to the second row, and so forth.

for link in links:
    # links = driver.find_elements(by='xpath', value='//a[@title="route detail"]')
    link.click() 
    time.sleep(2)
    driver.back()

The code above works for the first run and then throws the error: Message: stale element reference: element is not attached to the page document

I tried to add various Wait, refresh etc etc but no success. I am using selenium 4.6.0. By the way, if I execute line by line outside of the for loop with Jupyter Notebook works.

I also added the find_element line inside the door loop but still doesn't work.

You need to redefine the links list every time you reload the page, or go back to it. If you still have issues after following these instructions, post your code including page url. — Barry the Platipus, Nov 22 '22 at 16:12
Adding this line `links = driver.find_elements(by='xpath', value='//a[@title="route detail"]')` inside the for loop (uncommenting line in the code) doesn't help. Is this what you meant? — Seve, Nov 22 '22 at 16:16
collecting the list again should be done after getting back to the main page — Prophet, Nov 22 '22 at 16:18

score 2 · Accepted Answer · answered Nov 22 '22 at 16:14

By navigating to another page all previously collected by Selenium web elements (they are actually references to a physical web elements) become no more valid since the web page is re-built when you open it again.
To make your code working you need to collect the links list again on the main page when you getting back.
So, this code should work:

links = driver.find_elements(by='xpath', value='//a[@title="route detail"]')
for index, link in enumerate(links):
    links[index].click() 
    # do what you want to do on the opened page
    driver.back()
    time.sleep(0.2)
    links = wait.until(EC.visibility_of_element_located((By.XPATH, '//a[@title="route detail"]')))

Selenium 4 (python): stale element reference during table web scraping

1 Answers1