1

I'm working on collecting data from an income statement data table in Yahoo Finance. There are collapsible/expandable sections within this table, which the web scraper can't seem to access. How do I retrieve data from the collapsed sections? Here's the link to the site I'm scraping from: https://finance.yahoo.com/quote/AMZN/financials?p=AMZN.

So far I've written a program that can scrape the visible sections of the income statement. I expected to retrieve the hidden data too, since it's all under the same div container that I'm iterating through.

new_url = "https://finance.yahoo.com/quote/AMZN/financials?p=AMZN"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}

new_page = requests.get(new_url, headers=headers)

new_soup = BeautifulSoup(new_page.content, "html.parser")
new_table = new_soup.find_all(class_="M(0) Whs(n) BdEnd Bdc($seperatorColor) D(itb)")

for elem in new_table:
    string = str(elem.text)
    print(string)
Mulak
  • 11
  • 1

0 Answers0