Scraping data from google finance using BeautifulSoup in python

Question

I'm trying to get data from google finance from this link like this:

url = "https://www.google.com/finance/historical?cid=4899364&startdate=Dec+1%2C+2016&enddate=Mar+23%2C+2017&num=200&ei=4wLUWImyJs-iuASgwIKYBg"
request = urllib.request.Request(url,None,headers)
response = urllib.request.urlopen(request).read()
soup = BeautifulSoup(response, 'html.parser')
prices = soup.find_all("tbody")
print(prices)

I'm getting an empty list. I have also tried alternates like using soup.find_all('tr') but still I can't retrieve data successfully.

edit:

headers={'Host': 'www.google.com',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Connection': 'keep-alive'
}

What is `headers`? If you just want the table, probably using `pandas` may be much more convenient. `pd.read_html(url)` would give you a list, in which the 2nd item should be the table. — mikeqfu, Mar 23 '17 at 17:52
Or try `soup = BeautifulSoup(response, 'lxml')`. (You may need to `pip install lxml` if you haven't yet got it in your site-packages directory.) — mikeqfu, Mar 23 '17 at 18:03
trying different option like using pandas as mentioned by @mikeqfu, using requests or using lxml — Ni3_k, Mar 23 '17 at 18:11

score 0 · Accepted Answer · answered Mar 23 '17 at 18:41

0

The problem was with html.parser. I instead used lxml and it worked. Also exchanged urllib with requests.

answered Mar 23 '17 at 18:41

Ni3_k

56
12

score 0 · Answer 2 · answered Feb 23 '23 at 14:33

Unfortunately there's no /historical URL route. Provided link leads to Google Finance Home page.

For Google Finance scraping, you can use Google Finance API from SerpApi. This is a paid API with a free plan that processes blocks and parses them on its backend.

Check SerpApi simple code in the online IDE.

from serpapi import GoogleSearch
import json, os

params = {
  "engine": "google_finance",      # serpapi parser engine. Or engine to parse ticker page: https://serpapi.com/google-finance-api
  "q": "GOOG:NASDAQ",              # parameter defines the query you want to search. It can be a stock, index, mutual fund, currency or futures
  "api_key": "..."                 # serpapi key, https://serpapi.com/manage-api-key
}

search = GoogleSearch(params)      # where data extraction happens
results = search.get_dict()        # JSON -> Python dict
 
summary = results["summary"]

print(json.dumps(summary, indent=2, ensure_ascii=False))

Output:

{
  "title": "Alphabet Inc Class C",
  "stock": "GOOG",
  "exchange": "NASDAQ",
  "price": "$94.93",
  "extracted_price": 94.93,
  "currency": "$",
  "price_movement": {
    "percentage": -0.021060184,
    "value": -0.019996643,
    "movement": "Up"
  },
  "extensions": [
    "Feb 15, 9:38:23 AM GMT-5",
    "USD",
    "NASDAQ"
  ]
}

There are also several blog posts such as: Scrape Google Finance Ticker Quote Data in Python, Scrape Google Finance Markets in Python and Web Scraping Google Finance Main Page in Python, which also shows a DIY parsing solution with code explanation.

Disclaimer, I work for SerpApi.

Scraping data from google finance using BeautifulSoup in python

2 Answers2