0
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
path_to_chromedriver = 'C:/Users/WIN7/AppData/Local/Programs/Python/Python37-32/chromedriver.exe'
browser = webdriver.Chrome(executable_path=path_to_chromedriver)
browser.get('https://shipped.com/shipping-containers-for-sale.php')
Country_Click = browser.find_element_by_xpath("//select[@name='country']")
all_options = Country_Click.find_elements_by_tag_name("option")
for option in all_options:
    print("Value is: %s" % option.get_attribute("value"))
    option.click()
    browser.implicitly_wait(20)
    state_click = browser.find_element_by_xpath("//select[@name='state']")
    all_options1 = state_click.find_elements_by_tag_name("option")
    for option in all_options1:
        print("Value is: %s" % option.get_attribute("value"))
        option.click()
        city_click = browser.find_element_by_xpath("//select[@name='city']")
        all_options2 = city_click.find_elements_by_tag_name("option")
        browser.implicitly_wait(20)
        for option in all_options2:
            print("Value is: %s" % option.get_attribute("value"))
            option.click()
            time.sleep(20)
            browser.implicitly_wait(20)
            rows = browser.find_element_by_tag_name('tbody').find_elements_by_tag_name('tr')
        states = []
        for row in rows:
            cells = row.find_elements_by_tag_name('td')
            SizeOfContainer = cells[0].text.replace('Double \n Doors', "")
            SizeOfContainer1 = SizeOfContainer.replace('HC', "")
            SizeOfContainer2 = SizeOfContainer1.replace('Double Doors', "")
            SizeOfContainer3 = SizeOfContainer2.replace('\n', "")
            SizeOfContainer4 = SizeOfContainer3.replace('DoubleDoors', "")
            Ranking = cells[1].text
            Price = cells[2].text
            DeliveryAvailable = cells[3].text

            Final = (SizeOfContainer4.replace("What's this?","") + "," + Ranking + "," + Price.replace(",", "|") + "," + DeliveryAvailable.replace(",", "|") + "\n")
            Output = Final.replace(
                #"United States,Alabama,Want More?,  10' \n to \n 53', Rare containers + common sizes. High-\ncube or standard, we've got you covered.,  Request A Quote »","")

            print(Output)

in the above code i get the stale exception error in the last for loop it runs for once but for the next value in the drop down i get the exception.I am trying to accomplish the task as the first drop-down describes the country, the second describes the states int he country and the third describes the cities in that state. I need to iterate i over all the cities in each state in each country but give me a stale exception error

as below:

Traceback (most recent call last):
  File "C:/Users/WIN7/.PyCharmCE2018.1/config/scratches/FinalShipped.py", line 24, in <module>
    print("Value is: %s" % option.get_attribute("value"))
  File "C:\Users\WIN7\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 143, in get_attribute
    resp = self._execute(Command.GET_ELEMENT_ATTRIBUTE, {'name': name})
  File "C:\Users\WIN7\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "C:\Users\WIN7\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 320, in execute
    self.error_handler.check_response(response)
  File "C:\Users\WIN7\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=67.0.3396.99)
  (Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 6.1.7601 SP1 x86_64)
codehacker
  • 320
  • 2
  • 3
  • 16
  • Hi, how does the html look like? – kratze Jul 06 '18 at 11:07
  • @kratze the link for html is too long and the url for the same is https://shipped.com/shipping-containers-for-sale.php – codehacker Jul 06 '18 at 11:37
  • so far as i can see it might be a problem to find the options on the page. the select box for cities is empty when you go on the page, it is only filled with the cities when you select a state / region before. i didnt check your whole code but i think you must firstly select a state / region and then you can only iterate through the city options. – kratze Jul 06 '18 at 11:52
  • @kratze i am selecting the state and the citi also gets selected but for the next city i get the stale element exception – codehacker Jul 06 '18 at 11:58
  • in the loop where you click to chose the city, try to set a sleep to 3-5 seconds. I notice on the website, the the select for cities is very slow, because it fetches data from a php file. In python you need to "import time" and use "time.sleep(5)" in the select city loop – kratze Jul 06 '18 at 12:01
  • @kratze still getting the same error because after i select the citi the page refreshes and i get the stale exception – codehacker Jul 06 '18 at 12:09
  • i think you cant just loop through all cities. you could create a list where you store all cities available for a state... then you loop through the length of that list and select first select options from the cities, wait until refresh, scrape all content from new page, continue select next city. in selenium you must change your markup for cities then, you must work with the css selector "li:nth-child(n)" instead of the n you insert the iterator from your loop. – kratze Jul 06 '18 at 12:13
  • @kratze if you could help me with the code snippet for the same to first store it in a list and then iterate over it .. will mean alot – codehacker Jul 06 '18 at 12:15
  • well i try to send you later something. i personaly would have used beautifulsoup for the scraping. i already tried something out, you can send the city value as a post to the php file and you get then the content back and there you can just parse the html with beautifulsoup. in my opinion much easier and faster. – kratze Jul 06 '18 at 12:25
  • If you are getting stale element exception you need to re-get your elements from the page after the refresh and start with those element. You cannot start with the same elements list/arrays when the page refreshes. As a remedy, you can put the re-getting the page elements everytime the loop starts and refresh the page. Just put the code that gets the elements in loop again and try. – PJAutomator Jul 06 '18 at 15:44

0 Answers0