How can I parse a dynamic page using Python?

Question

I am using Ghost and BeautifulSoup to parse a HTML page. The problem that I have, is that the content of this page is dynamic (created with angularJS). At the beginning the html only shows something like "please wait! page loading". After a few seconds the content of the html appears. Using Ghost and BeatifulSoup I just get the HTML code of the loading page whith only 2 small divs. The URL stays the same. Is there a possibility to wait until the "real" content is loaded?

Answer 1

Load the page in a real browser (headless like PhantomJS is also an option) automated by selenium, wait for the desired contents to appear, get the .page_source and pass it to BeautifulSoup:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.PhantomJS()
driver.get("your url here")

# waiting for the page to load - TODO: change
wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.ID, "content")))

data = driver.page_source
driver.close()

soup = BeautifulSoup(data, "html.parser")

Answer 2

Use phantomjs to open the page. Save it as a local file using phantomjs File System Module Api. Later use this local file handle to create BeautifulSoup object and then parse the page. See http://www.kochi-coders.com/2014/05/06/scraping-a-javascript-enabled-web-page-using-beautiful-soup-and-phantomjs/