Find element in desplegable list after doing browser back

Question

I'm scraping a web with the following structure

Menu1
  Submenu1
    Event1
    Event2
    ...
  Submenu2
    Event n
    ...
  ...
Menu2
  ....

Where to have access to Submenu you have to click on Menu, so as to expand the node and make submenu list visible, and to have access to Event list the same, you need to click on the corresponding submenu node. Once you get the event, you click on it and it goes to another page. After you scrape some, you go back to the previous page and scrape the next event. The code would be as follows

browser=webdriver.Chrome()
browser.get(url)

Menu1=browser.find_element_by_xpath('some expression')
Menu1.click()

submenu=Menu1.find_elements_by_xpath('some other expression')
for sub in submenu:
   event=sub.find_elements_by_xpath('expression here')
   for ev in event:
     event.click()
     Some scraping
     'Go back to previous page'
     browser.execute_script("window.history.go(-1)")

After doing the first iteration when trying to do the second for Event2, I get a NoSuchElementException, basically because after going back list are not expanded and therefore not all objects are visible to keep on with the for looping.

Is there any way that suits the code to fix this? My guess is that if I click Menu1 again, all Submenu elements will be visible so that list will be once again available. Afterwards, if I click on sub then the Events list will be available as well.

Eager to read your suggestions Thanks

score 0 · Answer 1 · answered Mar 08 '18 at 14:25

So, in esence, what has come to my mind as the solution is generating several list for each menu, with the text of the node. Iterate in for loops instead of nodes, loops of the node names that are unchanging. Then, for each node name, loop through the object until finding it, and click on that particular item.

Then repeat until you've reached all nodes.

Find element in desplegable list after doing browser back

1 Answers1