4

I know this question was asked before but none of the proposed solutions work in my case. I am trying to web-scrape a page of results but the problem is that 95% of div tags contain only class names that are dynamically changing. My code works for several hours using Xpath or a whole day but at some point class names slightly change and my program breaks. Xpath obviously does not work since it changes with the content. I wanted to make sure I fully understand the limitations of web-scraping using Selenium when I have limited options in terms of selecting tags. Is there any other solution that could work in situation like mine where I only have a page of results full of div tags and dynamically named classes?

detrraxic
  • 156
  • 6
  • sometimes you can find the data you need in the script tags, or by accessing an API that renders that data. Can I ask what the url is? – chitown88 Mar 19 '19 at 13:11
  • 2
    You need to anchor your locators using elements that have unique, static attributes or find elements based on their relationship to elements that contain static text. If you put a link to the page (or the relevant HTML) and some example elements that you want to locate, we can help create locators to find them. – JeffC Mar 19 '19 at 13:23
  • hi all, thanks for the answers. This is the example link: https://www.mozio.com/search?start_address=MIA&end_address=1750%20NW%20107th%20Ave%2C%20Doral%2C%20FL%2033172%2C%20USA&pickup_datetime=03%2F28%2F2019%2001%3A00%20PM&mode=one_way&num_passengers=1&currency=USD&return_pickup_datetime=03%2F30%2F2019%2001%3A00%20PM. I am trying to scrape results (vehicle categories, company names, image urls, price, etc). – detrraxic Mar 22 '19 at 12:17

0 Answers0