Getting "StaleElementReferenceException" when accessing options in drop down menu

Question

In Python and Selenium, I'm populating a form, submitting it, then scraping the resulting multi-page table that appears on the page underneath the form. After I scrape every page of this table, I reset the form and attempt to repopulate the form. However, a drop down menu is tripping up the code.

I've tried to make the driver wait for the drop down menu to reappear after I reset the form, but this doesn't help. I still receive the StaleReferenceElementException error on the if option.text == state line:

 StaleElementReferenceException: Message: The element reference of <option> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

How do I submit the form over and over for different options within the drop down menu?

states = ['Alabama', 'Alaska', 'Arizona', 'Arkansas', 'California', 
          'Colorado', 'Connecticut', 'Delaware', 'District of Columbia', 
          'Florida', 'Georgia', 'Hawaii', 'Idaho', 'Illinois', 'Indiana',
          'Iowa', 'Kansas', 'Kentucky', 'Louisiana', 'Maine', 'Maryland',
          'Massachusetts', 'Michigan', 'Minnesota', 'Mississippi', 'Missouri',
          'Montana', 'Nebraska', 'Nevada', 'New Hampshire', 'New Jersey', 
          'New Mexico', 'New York', 'North Carolina', 'North Dakota', 'Ohio',
          'Oklahoma', 'Oregon', 'Pennsylvania', 'Rhode Island', 'South Carolina',
          'South Dakota', 'Tennessee', 'Texas', 'Utah', 'Vermont', 'Virginia',
          'Washington', 'West Virginia', 'Wisconsin', 'Wyoming']

# Construct browser and link
browser = webdriver.Firefox(executable_path='/usr/local/bin/geckodriver')
url = 'https://myaccount.rid.org/Public/Search/Member.aspx'
ignored_exceptions = (StaleElementReferenceException,)

# Navigate to link
browser.get(url) 

try:
    # For each state
    for state in states:
        print('Searching ' + state)
        # Set category and select state menu
        category = Select(browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$categoryDropDownList'))
        category.select_by_value('a027b6c0-07bb-4301-b9b5-1b38dcdc59b6')
        state_menu = Select(WebDriverWait(browser, 10, ignored_exceptions=ignored_exceptions).until(EC.presence_of_element_located((By.ID, 'FormContentPlaceHolder_Panel_stateDropDownList'))))
        options = state_menu.options

        for option in options:
            if option.text == state:
                state_menu.select_by_value(option.get_attribute('value'))
                browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$searchButton').click()

                # Scrape the first page of results
                results = []
                curr_page = 1
                onFirstPage = True
                scrape_page(curr_page)

                # Reset form
                browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$resetButton').click()
                break
finally:
    pass

supputuri · Accepted Answer · 2020-02-05T20:40:12.153

0

The moment you select the option, element references will update and you can't use the older references. Reason you are getting the exception is, you are trying to get the attribute from the option which no longer valid.

Rather using the iteration, I would use the xpath to select the option as shown below

        state_menu = WebDriverWait(browser, 10, ignored_exceptions=ignored_exceptions).until(EC.presence_of_element_located((By.ID, 'FormContentPlaceHolder_Panel_stateDropDownList')))
        #options = state_menu.options <== replace this line with below line
        option = state_menu.find_element_by_xpath("//option[.='" + state + "']")

        #for option in options: <== remove this line
            # if option.text == state: <== remove this
        option.click()
        browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$searchButton').click()

        # Scrape the first page of results
        results = []
        curr_page = 1
        onFirstPage = True
        scrape_page(curr_page)
        # Reset form
        browser.find_element_by_name('ctl00$FormContentPlaceHolder$Panel$searchButtonStrip$resetButton').click()

edited Feb 05 '20 at 20:40

answered Feb 05 '20 at 18:42

supputuri

13,644
2
21
39

Thanks! Now looking into why this returns `AttributeError: 'Select' object has no attribute 'find_element_by_xpath'` – brienna Feb 05 '20 at 20:02
That's because you are returning `state_menu` object rather element. I updated the answer (first line) to return the element rather the object. That should resolve the issue. – supputuri Feb 05 '20 at 20:21
That returns `AttributeError: 'FirefoxWebElement' object has no attribute 'select_by_value'` at `state_menu.select_by_value(option.get_attribute('value'))`. Wouldn't we need to Select the drop down menu? – brienna Feb 05 '20 at 20:38
My bad, I overlooked that step where you select the option. Updated the answer to click on the option. – supputuri Feb 05 '20 at 20:40
Unfortunately this code does not work. There are too many errors that keep arising after I fix each one. – brienna Feb 05 '20 at 23:03
However, it did fix the "StaleElementReferenceException" error so I'll mark it as accepted. – brienna Feb 05 '20 at 23:16

Getting "StaleElementReferenceException" when accessing options in drop down menu

1 Answers1