0

I'm new python user and I want to scrape data from this website: https://www.telerad.be/Html5Viewer/index.html?viewer=telerad_fr

My problem is that the data are dynamically generated. I read few possibilities to fix but none is satisfying. With selenium I need a name or Xpath to click on button but here there is nothing. 

import requests
from lxml import html

page = requests.get('https://www.telerad.be/Html5Viewer/index.html?viewer=telerad_fr')
tree = html.fromstring(page.content)

cities = tree.xpath('//*[@id="map-container"]/div[6]/div[2]/div/div[2]/div/div/div[1]/div/p[1]/text()[2]')


print('Cities: ', cities)
orde
  • 5,233
  • 6
  • 31
  • 33
félicien louyet
  • 3
  • 1
  • you can't get it with requests/urllib + Beautifulsoup/lxml because they don't run JavaScript. You have to use `Selenium` to get it. If Javascript read data from server then you can try to find this url in DevTools in Chrome/Firefox and use this url with `requests` to get it. Mostly JavaScript gets data in JSON format which can be easily converted to python's dictionary/list and you don't have to scrape HTML. – furas May 07 '19 at 15:52
  • what button do you have to click ? if you want help then you have to add all information in question - we can't read in your mind. – furas May 07 '19 at 15:54
  • I want to click on little orange dot on the map to get data :) – félicien louyet May 07 '19 at 16:08

1 Answers1

0

There actually IS an xpath to click on the buttons:

//*[@id='0_layer']/*[@fill]

Here, try this (selenium):

dotList = driver.find_elements_by_xpath("//*[@id='0_layer']/*[@fill]")
for dot in dotList:
    dot.click()
    cities = driver.find_element_by_xpath("//div[@data-region-name='NavigationMapRegion']//p[1]")
    print("Cities: ", cities.text)
    closeBtn = driver.find_element_by_xpath("//*[@class='panel-header-button right close-16']")
    closeBtn.click(); #the modal can intercept clicks on some dots, thats why we close it here after extracting the info we need.

this code clicks (or at least tries to, if no StaleElementExceptions occur) all the orange dots on the map, and print the "Cities" content (based on your Xpath).

If anyone finds an error in the code, please edit this answer, i wrote this on notepad++.

Valga
  • 459
  • 2
  • 7