Selenium returning to previous page in a for loop

Question

I'm making a scraper to scrape a betting website for its data, this is an example piece of code that will scrape the event, teams and odds. The idea is to iterate through all the competitions that are available for the sport. I am able to locate the links and click on the first one, then I am also able to scrape the data and return it in a list (to later be put into a database). My issue is I am not able to return to the previous page to click the next link, I get an error when trying to do this. Here is my code:

driver = webdriver.Chrome(Path)
driver.get("https://www.neds.com.au/sports/table-tennis/")
a = []

links = driver.find_elements_by_class_name("matches-filter__link")

for l in links:
    
    l.click()
    tt_matches = driver.find_elements_by_class_name("sport-event-card")
    for match in tt_matches:
        Match = match.find_element_by_css_selector(".sports-event-title__name-text").text
        a.append(Match)
        Teams = match.find_elements_by_css_selector(".price-button")
        for team in Teams:
            team_name = team.find_element_by_css_selector(".price-button-name").text
            team_odd = team.find_element_by_css_selector(".price-button-odds-price span").text
            a.append(team_name)
            a.append(team_odd)
    driver.back()
    time.sleep(2)

driver.quit()

and it returns this error:

raise exception_class(message, screen, stacktrace)

StaleElementReferenceException: stale element reference: element is not attached to the page document (Session info: chrome=84.0.4147.105)

How can I fix this, I think the issue is with the driver.back(), I have also tried l.back() and it doesn't work still.

I think you need to redefine `links` after you've come back to the page. see [this](https://www.softwaretestingmaterial.com/stale-element-reference-exception-selenium-webdriver/) for more info. — Beek, Aug 05 '20 at 07:09

score 3 · Accepted Answer · answered Aug 05 '20 at 07:16

In this case, you need to collect URLs and then navigate to them. you can use following code:

driver.get("https://www.neds.com.au/sports/table-tennis/")
time.sleep(5)
a = []

links = driver.find_elements_by_class_name("matches-filter__link")
urls = [l.get_attribute('href') for l in links]
for u in urls:
    print(u)
    driver.get(u)
    tt_matches = driver.find_elements_by_class_name("sport-event-card")
    for match in tt_matches:
        Match = match.find_element_by_css_selector(".sports-event-title__name-text").text
        a.append(Match)
        Teams = match.find_elements_by_css_selector(".price-button")
        for team in Teams:
            team_name = team.find_element_by_css_selector(".price-button-name").text
            team_odd = team.find_element_by_css_selector(".price-button-odds-price span").text
            a.append(team_name)
            a.append(team_odd)
    time.sleep(2)

print(a)
driver.quit()

I'd like to understand where the error comes from and why? I'm running into the same error but it's when I'm doing a `driver.execute_script()` and I have found nothing close to answering it. It seems OK deep into urllib3 AFAICT, but the response from selenium is: ```{"value":{"error":"stale element reference","message":"stale element reference: stale element not found\\n (Session info: chrome=84.0.4147.105)","stacktrace":"#0 0x561877802d99 \\u003Cunknown>\\n"}}'``` The command is run many times before failing. Is there some limit or counter? — iJames, Aug 21 '20 at 01:47

Selenium returning to previous page in a for loop

1 Answers1