I'm working on collecting data from an income statement data table in Yahoo Finance. There are collapsible/expandable sections within this table, which the web scraper can't seem to access. How do I retrieve data from the collapsed sections? Here's the link to the site I'm scraping from: https://finance.yahoo.com/quote/AMZN/financials?p=AMZN.
So far I've written a program that can scrape the visible sections of the income statement. I expected to retrieve the hidden data too, since it's all under the same div container that I'm iterating through.
new_url = "https://finance.yahoo.com/quote/AMZN/financials?p=AMZN"
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}
new_page = requests.get(new_url, headers=headers)
new_soup = BeautifulSoup(new_page.content, "html.parser")
new_table = new_soup.find_all(class_="M(0) Whs(n) BdEnd Bdc($seperatorColor) D(itb)")
for elem in new_table:
string = str(elem.text)
print(string)