Intermittent Stale Element Reference Exception

Question

I am writing a python selenium script to scrape data from a website. The script reads the employee identification number, name and date of birth from a CSV file and copies it to a Dictionary & then enters data from each line into a form and then clicks a submit button. The next page that appears asks for the years of employment history which defaults to one year which is fine for my use case so the only thing the script does on this page is click the search button. The script works consistently for approximately 20 to 35 employees. But at some point the script will fail with the following stale element reference exception:

Message: The element reference of is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

The relevant HTML for the Search button is:

<div data-container="" class="btn-label OSInline">Search</div>
</button>
</div>
</div>
</div><div data-container="" style="text-align: center;" id="b8-ResetFilters2"><a data-link="" href="#" style="width: auto;"><span data-expression="" style="width: auto;">Reset Search</span></a>
</div>
</div>
</div>
<div

The relevant Python Selenium code is (GO DOWN TO "*****" TO SEE WHERE I THE SEARCH BUTTON IS):

# Open MOCemployees.csv as readable file & then convert to a Python Dictionary
with open('MOCemployees.csv', 'r') as MOCemployees.:
    MOCemployees.Dict = csv.DictReader(MOCemployees.)

    for line in MCpatientsDict:
        MOCEmployeeNumber = (line['MCID'])
        LastName = (line['LastName'])
        FirstName = (line['FirstName'])
        DOB = (line['DOB'])

# Explicit Wait for MOCEmployeeNumberInput then click on it and enter MOCEmployeeNumber
        MOCEmployeeNumberInput = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, 'b7-MOCEmployeeNumberInput')))
# Locate LastNameInput and click on it and enter LastName
        LastNameInput = driver.find_element(By.ID, 'b7-LastNameInput')
        LastNameInput.click()
        LastNameInput.send_keys(LastName)
# Locate FirstNameInput and click on it and enter FirstNameI
        FirstNameInput = driver.find_element(By.ID, 'b7-FirstNameInput')
        FirstNameInput.click()
        FirstNameInput.send_keys(FirstName)
# Locate DOBInput and click on it and enter DOB
        DOBInput = driver.find_element(By.ID, 'b7-DOBInput')
        DOBInput.click()
#DOBInput.send_keys('YYYY-MM-DD')
        DOBInput.send_keys(DOB)

# Explicit Wait for Submit Button & click on it
        SubmitButton = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button.margin-top-m')))
        SubmitButton.click()

# *****Explicit Wait for Search Button & click on it to accept default 1 year of data *****THIS BUTTON IS NOT CLICKED INTERMITTENTLY AND THE STALE ELEMENT REFERENCE EXCEPTION OCCURS AT THIS POINT*****
        SearchButton = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.btn-label')))
        SearchButton.click()
#Wait
        time.sleep(3)

# print MOCID, NAME & then the value of the value attribute 
        RemainDeductible = (WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "b8-b36-Input_RemainAmtYr1"))).get_attribute("value"))

#Save MOCID, Name & Remaining Due to a CSV file
        POutput = ','.join((MOCEmployeeNumber, LastName, FirstName, DOB, RemainDeductible)) + '\n'

#Locate Change Employee Link & click on it
        SelectButton = driver.find_element(By.CSS_SELECTOR, '#b7-b3-Column4 > div:nth-child(1) > a:nth-child(1) > span:nth-child(1)')
        SelectButton.click()

How can I prevent this stale element reference exception?

The complete error message is Message: The element reference of
is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed — Auditor, Mar 16 '22 at 16:39
does this `btn-label OSInline` and `.btn-label` where you are getting exception represent the same element ? — cruisepandey, Mar 16 '22 at 16:40
@cruisepandey Good question. I double checked and yes ".btn-label" is the CSS Selector for "btn-label OSInline". — Auditor, Mar 16 '22 at 20:40

score 0 · Answer 1 · answered Mar 16 '22 at 16:46

0

Maybe the element is not accessible at the very first attempt in the HTML DOM.

You can try to perform click again in a loop like this:

Code:

# *****Explicit Wait for Search Button & click on it to accept default 1 year of data *****THIS BUTTON IS NOT CLICKED INTERMITTENTLY AND THE STALE ELEMENT REFERENCE EXCEPTION OCCURS AT THIS POINT*****
attempts = 0
while attempts < 2:
    try:
        SearchButton = WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.btn-label')))
        SearchButton.click()
        break
    except StaleElementReferenceException as e:
        print('')
       # print('Could not click', e.msg)
    attempts = attempts  + 1

You can probably try with while attempts < 5:

answered Mar 16 '22 at 16:46

cruisepandey

28,520
6
20
38

I appreciate your effort but I am getting the same error with the loop while attempts < 2: (I also increased it to 5 & then 15 with the same results). I thought that you were on to the right track because when the "stale element reference exception" does occur, I am still able to manually click on the link without any issue. – Auditor Mar 16 '22 at 19:26
Are you sure that this locator is unique `.btn-label` ?Also, could you try with `div.btn-label.OSInline`. `while attempts < 5:` should be sufficient though. Also before `attempts = 0` put a `time.sleep(5)` for debugging purpose and see if that helps – cruisepandey Mar 17 '22 at 13:00
I was able to change the code to `div.btn-label.OSInline`. Unfortunately it still failed on the 28th employee. When I added the `time.sleep(5)` before `attempts = 0`, **IT FINALLY WORKED** and went through all of the employees in the dictionary/CSV file. But I don't know how this helps us with debugging. What should I be looking for now? – Auditor Mar 18 '22 at 20:48
I reduced the `time.sleep(5)` to `time.sleep(2)` and it ***still works***. But I still don't know _why_ this helps us with debugging. What should I be looking for now? – Auditor Mar 19 '22 at 01:12
1

I got it, the issue is with the element rendered, when you put `time.sleep(2)` it's a hardcoded wait which will stop the thread for 2 secs and in this duration element would have rendered properly. There's no harm having `time.sleep(2)` if you observe a consistency with it. – cruisepandey Mar 19 '22 at 05:34

Intermittent Stale Element Reference Exception

1 Answers1