Python Selenium - StaleElementReferenceException: stale element not found

Question

I am trying to click on the next button at the bottom of the page to access the following page as in the image below:

However, when I reach page 3 I receive the following error:

selenium. common. exceptions. StaleElementReferenceException: Message: stale element reference: stale element not found

from the code line

next_button = wait.until(EC.element_to_be_clickable(span_element[1])).click()

Overall, the code I am using for this looks like

region_content = driver.find_element(By.CLASS_NAME, "pager__item--next")
span_element = region_content.find_elements(By.CSS_SELECTOR, "a > span")  
next_button = wait.until(EC.element_to_be_clickable(span_element[1])).click()
       

and the HTML code where the button can be found is the following:

<li class="pager__item pager__item--next">
          <a href="?field_headquarters_of_company_target_id=All&amp;ajax_page_state%5Btheme%5D=bootstrap&amp;ajax_page_state%5Blibraries%5D=addtoany/addtoany%2Casset_injector/css/css_injector%2Cbootstrap/popover%2Cbootstrap/tooltip%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/drupal.autocomplete%2Ccore/html5shiv%2Cextlink/drupal.extlink%2Cgoogle_analytics/google_analytics%2Csuperfish/superfish%2Csuperfish/superfish_hoverintent%2Csuperfish/superfish_smallscreen%2Csuperfish/superfish_style_white%2Csuperfish/superfish_supersubs%2Csuperfish/superfish_supposition%2Csuperfish/superfish_touchscreen%2Csystem/base%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.ajax%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cviews/views.module%2Cwebform_bootstrap/webform_bootstrap&amp;ajax_page_state%5Btheme_token%5D=&amp;_wrapper_format=drupal_ajax&amp;title=&amp;field_company_category_primary_target_id&amp;field_market_cap_jul302023_value=&amp;field_company_website_uri=&amp;page=3" title="Go to next page" rel="next">
            <span class="visually-hidden">Next page</span>
            <span aria-hidden="true">Next ›</span>
          </a>
        </li>

I have tried with a try/except statement to refresh the page in case of stale element but still does not work. What is a bit puzzling to me is that to go from page 1 to 2 and from 2 to 3 there is no error.

Would anyone be willing to help out?
Thank you

Answer 1

You already have a working Selenium setup, so :

[...]
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
[...]
wait = WebDriverWait(driver, 25)
url = 'https://www.value.today/'
driver.get(url)
while True:
    wait.until(EC.element_to_be_clickable((By.XPATH, '//a[@rel="next"]'))).click()
    t.sleep(1)
    print('clicked for next page')

This will click to go to the next page, and print out in terminal:

clicked for next page
clicked for next page
clicked for next page
clicked for next page
clicked for next page
[..]

EDIT: In case you're in one of the regions requiring cookie consent, you will need to handle the cookie popup. Here is the updated code:

[..]
wait = WebDriverWait(driver, 25)

url = 'https://www.value.today/'
driver.get(url)

try:
    wait.until(EC.element_to_be_clickable((By.XPATH, '//p[text()="Consent"]'))).click()
except Exception as e:
    print('no cookie button!')
    
while True:
    try:
        items_list = wait.until(EC.presence_of_all_elements_located((By.XPATH, '//div[@class="item-list"]//li[@class="row well clearfix"]')))
        for item in items_list:
            comp_title = WebDriverWait(item, 25).until(EC.element_to_be_clickable((By.XPATH, './/h2')))
            employee_count = WebDriverWait(item, 25).until(EC.element_to_be_clickable((By.XPATH, './/span[contains(text(), "Employee Count: ")]/following-sibling::span')))
            print(comp_title.text, employee_count.text)

        wait.until(EC.element_to_be_clickable((By.XPATH, '//a[@rel="next"]'))).click()
        t.sleep(1)
        print('clicked for next page')
    except Exception as e:
        print('error, stopping now...')
        break

Result in terminal:

APPLE 164,000
MICROSOFT CORPORATION 220,000
SAUDI ARABIAN OIL COMPANY (Saudi Aramco) 79,800
ALPHABET 190,234
AMAZON.COM 1,541,000
NVIDIA CORPORATION 26,196
TESLA 127,855
META PLATFORMS 71,469
BERKSHIRE HATHAWAY 383,000
TAIWAN SEMICONDUCTOR MANUFACTURING COMPANY (TSMC) 73,090
clicked for next page
VISA 26,500
UNITEDHEALTH GROUP 380,000
[..]

You can find Selenium documentation here.

Answer 2

What browser you use? From my experience, it's best to click elements by xpath.

Try something like:

while True:
    try:
        driver.find_element('xpath', 'some_xpath').click()
        break
    except:
        pass