0

I am trying web scrape the list from a real estate website for example https://nz.hougarden.com/house-search/type_sold_category_9_district_waitakere-city-230_order_0_page_2

I see the listings are within class called column col-3 col-lg-12 col-xl-6 col-4x-4

page_soup = soup(webpage,"html.parser")
containers = page_soup.findAll("div",{"class":"column col-3 col-lg-12 col-xl-6 col-4x-4"})

len(containers) returns 0?

is it because there is a hidden container with 0 data so it returns empty?

<div class="column col-3 col-lg-12 col-xl-6 col-4x-4" style="display: none;"><!----></div>

If so how do I get around this or can someone please help me to make it work?

Thanks

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Alex1991
  • 13
  • 1
  • 7
  • 1
    Please do not share information as images unless absolutely necessary. See: https://meta.stackoverflow.com/questions/303812/discourage-screenshots-of-code-and-or-errors. Have you checked that the data you need is not dynamically generated? – AMC Feb 08 '20 at 02:56
  • The content is rendered dynamically with Javascript. Requests only gets you the raw HTML that does not contain the listings. You need to use a headless browser like Selenium. – petezurich Feb 09 '20 at 18:41
  • See also this: https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python – petezurich Feb 09 '20 at 18:42
  • @AMC Sorry I have removed the image now, thanks for your response. – Alex1991 Feb 09 '20 at 20:54
  • 1
    @petezurich Thank you for your help, I will look into how to web-scrape javascript pages. Cheers – Alex1991 Feb 09 '20 at 20:56

0 Answers0