1

I'm currently trynig to https://www.wolframalpha.com/input/?i=AAPL+price and get to the h2 tag that includes the text "Market Data".

I am doing this using BeautifulSoup by finding all h2 tags and checking h2.text for "Market Data".

However, the problem is the soup version of this url's text does not include any h2 tags. This contrasts with when I click "inspect element" and find h2s in Chrome. Why can't I see h2s when I do soup(url.text)? Does that mean its not a static page or something? How then can I scrape this h2 tag?

Essam
  • 157
  • 1
  • 2
  • 12
  • The scraped data doesn't seem to have an ng-include tag. There are ng-apps and ng-controllers, but I don't see any links. – Essam Mar 12 '17 at 22:40

1 Answers1

1

The actual source is from the URL linked in the ng-include tag (documentation). A detailed answer on how to account for angularJS source, which is not static but dynamically loaded, can be found here

Community
  • 1
  • 1
Preston Garno
  • 1,175
  • 10
  • 33
  • Ok, I will check out the documentation. – Essam Mar 12 '17 at 22:40
  • @Essam I also added link explaining how to go about this using BeautifulSoap, good luck – Preston Garno Mar 12 '17 at 22:42
  • So I've done what you suggested but the request that renders the part I'm interested doesn't really have a pattern to it (its just a request with a long number). Also the part turns out to be a gif and not a text. Is Selenium my only option at this point? – Essam Mar 13 '17 at 01:02