1

I am trying to get the value of VIX from a webpage.

The code I am using:

raw_page = requests.get("https://www.nseindia.com/live_market/dynaContent/live_watch/vix_home_page.htm").text
soup = BeautifulSoup(raw_page, "lxml")
vix = soup.find("span",{"id":"vixIdxData"})
print(vix.text)

This gives me:

' '

If I see vix,

<span id="vixIdxData" style=" font-size: 1.8em;font-weight: bold;line-height: 20px;">/span>

On the site the element has text,

<span id="vixIdxData" style=" font-size: 1.8em;font-weight: bold;line-height: 20px;">15.785/span>

The 15.785 value is what I want to get by using requests.

Keyur Potdar
  • 7,158
  • 6
  • 25
  • 40
Sid
  • 3,749
  • 7
  • 29
  • 62

2 Answers2

1

The data you're looking for, is not available in the page source. And requests.get(...) gets you only the page source without the elements that are dynamically added through JavaScript. But, you can still get it using requests module.

In the Network tab, inside the developer tools, you can see a file named VixDetails.json. A request is being sent to https://www.nseindia.com/live_market/dynaContent/live_watch/VixDetails.json, which returns the data in the form of JSON.

enter image description here

You can access it using the built-in .json() function of the requests module.

r = requests.get('https://www.nseindia.com/live_market/dynaContent/live_watch/VixDetails.json')
data = r.json()
vix_price = data['currentVixSnapShot'][0]['CURRENT_PRICE']
print(vix_price)
# 15.7000
Keyur Potdar
  • 7,158
  • 6
  • 25
  • 40
0

When you open the page in a web browser, the text (e.g., 15.785) is inserted into the span element by the getIndiaVixData.js script.

When you get the page using requests in Python, only the HTML code is retrieved and no JavaScript processing is done. So, the span element stays empty.

It is impossible to get that data by solely parsing the HTML code of the page using requests.

SergiyKolesnikov
  • 7,369
  • 2
  • 26
  • 47
  • @Sid you may find these useful: https://stackoverflow.com/questions/43731197/evaluating-javascript-code-in-an-html-string-using-python https://stackoverflow.com/questions/16375251/evaluate-javascript-on-a-local-html-file-without-browser – SergiyKolesnikov Mar 07 '18 at 11:10