How to visit multiple links with selenium

Question

i'm trying to visit multiple links from one page, and then go back to the same page.

links = driver.find_elements(By.CSS_SELECTOR,'a')
    
for link in links:
     link.click() # visit page
     # scrape page 
     driver.back() # get back to previous page, and click the next link in next iteration

The code says it all

Answer 1

The logic in your code should work, however you might want to add a sleep in between certain actions, it makes a difference when scraping.

import time

and then add time.sleep(seconds) where it matters.

Answer 2

By navigating to another page all collected by selenium web elements (they are actually references to a physical web elements) become no more valid since the web page is re-built when you open it again. To make your code working you need to collect the links list again each time. This should work:

import time
links = driver.find_elements(By.CSS_SELECTOR,'a')
    
for i in range(len(links)):
    links[i].click() # visit page
    # scrape page 
    driver.back() # get back to previous page, and click the next link in next iteration 
    time.sleep(1) # add a delay to make the main page loaded
    links = driver.find_elements(By.CSS_SELECTOR,'a') # collect the links again on the main page

Also make sure all the a elements on that page are relevant links. Since this may not be correct