How to get html that I see in inspect element?

Question

I'm programming a web-scraper app with python. The website I want to scrape data use JS. How can I get the source that I see in inspect element?

score 1 · Answer 1 · answered May 31 '14 at 14:31

1

With javascript pycurl will not work, you need Selenium to get the stuff you need.

import selenium
driver = selenium.webdriver.Firefox()
driver.get("your_url")

Make sure you have Firefox (or another browser selenium supports) installed.

answered May 31 '14 at 14:31

Johan Rensink

I want the page html after js do it's function! – Bardia Heydari May 31 '14 at 14:52
Yeah that's what this will get you, either with driver.page_source after a waiting period [(check here)](http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp). Or, if the website is really unwilling, by using save_screenshot and OCR (that you probably have to write yourself). – Johan Rensink Jun 01 '14 at 06:35

1 Answers1