StaleElementException with Python

Question

I am trying to extract tables from dropdown menus but I keep on getting an exception error that the element is not attached to the document in the last loop. I think it is to do with the fact that the page refreshes when clicking on submit.

My code:

###Purpose: To retrieve gp election data 

import time
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

## wesbsite details
driver = webdriver.Chrome(executable_path = '/usr/local/bin/chromedriver')
driver.get('http://wbsec.gov.in/results/panchayat_election_detailed_result?election_year=2013')
driver.maximize_window()

## dropdown menu 
elements=driver.find_elements_by_tag_name("input")
for element in elements:
    if element.get_attribute("value")=="PANCHAYAT SAMITY":
        element.click()
        districts = Select(driver.find_element_by_id('zilla_parishad'))
        for district in districts.options:
            district.click()
            time.sleep(1)
            blocks=Select(driver.find_element_by_id('panchayat_samity'))
            for block in blocks.options:
                 block.click()
                 time.sleep(1)
                 poles=Select(driver.find_element_by_id('election_date'))
                 for pole in poles.options:
                     pole.click()
                     time.sleep(1)
                     test = driver.find_element_by_xpath("//input[@type='submit']").click()
                     time.sleep(5)

The error occurs in this line:

driver.find_element_by_xpath("//input[@type='submit']").click()

I think the issue is that the page does not have enough time to load. I have tried to increase the sleep time as well as use WebDriverWait but I get the same error each time.

I already asked you what was the expected output. You dint reply. Do you need data like for `PANCHAYAT SAMITY` which are the `zilla_parishad`? Or something else. — pmadhu, Sep 17 '21 at 09:58
Yes, I am trying to get the data for the PANCHAYAT SAMITY wise. — ak7880, Sep 17 '21 at 10:03
I have added a code. Check once. Do let me know if that is not what you are asking for. — pmadhu, Sep 17 '21 at 10:16
To clarify this not the same as zilla_parishad. First you select the zilla_parishad (district) then the Panchayat Samity (Block). — ak7880, Sep 17 '21 at 10:22

score 0 · Answer 1 · answered Sep 17 '21 at 10:02

0

The problem is not that it does not have time to load but rather the element is no longer fresh and accessible

You may want to try incorporate a way to refresh the page when receiving this error in order to have this element once more.

Another way to do this could be to save the text which you are using to find the elements each time you are searching rather than saving the physical element to search each loop.

i.e:

for i in range(len(driver.find_elements_by_tag_name("input"))):
    if driver.find_elements_by_tag_name("input")[i].get_attribute("value")=="PANCHAYAT SAMITY":
        driver.find_elements_by_tag_name("input")[i].click()

answered Sep 17 '21 at 10:02

Kwsswart

529
1
7
20

I tried this but I get the same error. – ak7880 Sep 17 '21 at 10:23
Try incorporating a refresh or work around this it is no clear cut solution but more intended to help guide you, each website is different to navigate, but if you are getting a StaleElementException this would be an indicator that the reference refered to is old and no longer available – Kwsswart Sep 17 '21 at 10:26

pmadhu · Answer 2 · 2021-09-17T11:55:34.990

What is happening is you are trying to select options from the dropdown with this for district in districts.options which has --Select-- also as option. And when you click on Get Results the page takes us somewhere else. We need to ignore the --Select-- option, so the for loop should be like this for i in range(1,len(districts)):

And after clicking on Get Results Selenium will not be able to recognize previous options, so we need to find those options inside the for loop again. Otherwise it throws Element not attached to the page exception.

At last, the Polling Date for some options might me nothing, so throws List of index out of range. So have put that in try block.

import time
from selenium import webdriver
from selenium.webdriver.support.ui import Select

driver = webdriver.Chrome(executable_path = 'path to chromedriver.exe')
driver.get('http://wbsec.gov.in/results/panchayat_election_detailed_result?election_year=2013')
driver.maximize_window()
driver.implicitly_wait(30)

driver.find_element_by_id("search_type_ps").click()
option1 = Select(driver.find_element_by_id("zilla_parishad")).options
for i in range(1,len(option1)):
    try:
        option1 = Select(driver.find_element_by_id("zilla_parishad")).options
        print("Options 1 :{}".format(option1[i].text))
        option1[i].click()
        time.sleep(2)
        options2 = Select(driver.find_element_by_id("panchayat_samity")).options
        opt2list = []
        for j in range(1,len(options2)):
            options2 = Select(driver.find_element_by_id("panchayat_samity")).options
            print("Options 2 : {}".format(options2[j].text))
            # opt2list.append(opt2.text)
            options2[j].click()
            time.sleep(2)
            options3 = Select(driver.find_element_by_id("election_date")).options
            try:
                for k in range(1,len(options3)):
                    options3 = Select(driver.find_element_by_id("election_date")).options
                    print("Options 3 : {}".format(options3[k].text))
                    ele_dates=options3[k].text
                    options3[k].click()
                    driver.find_element_by_xpath("//input[@name='submit']").click()
                    time.sleep(1)
            except:
                pass
    except:
        driver.get('http://wbsec.gov.in/results/panchayat_election_detailed_result?election_year=2013')
        driver.find_element_by_id("search_type_ps").click()

driver.quit()

Thanks a lot for the code. The iteration works for a while and I can see that the page loads but I get the following error: NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"[id="panchayat_samity"]"} — ak7880, Sep 17 '21 at 11:36
@ak7880 - That's because few of the options have `No results found`. Put the whole thing in a try block. If exception happens do everything from beginning. I have updated the code for the same. — pmadhu, Sep 17 '21 at 11:57
I mean, Do everything from beginning but with other options. — pmadhu, Sep 17 '21 at 12:05
The code works perfectly ! In the next step I am trying to extract all the tables generated so i assume I just add that in the try block ? — ak7880, Sep 17 '21 at 12:31
@ak7880 - Yes that would be correct. The code to extract tables would come after the `driver.find_element_by_xpath("//input[@name='submit']").click()` within for loop. — pmadhu, Sep 17 '21 at 12:38

StaleElementException with Python

2 Answers2