Python, requests webscraping - NSE India gives emtpy list

Question

I am trying to use requests to scrape data form NSE to find top gainers on Indian stock market. I have used this same method on yahoo finance and it worked but here, I keep getting the empty list as the result.

Here is my code:

import requests
from lxml import html
import urllib3


urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

resp = requests.get('https://www.nseindia.com/market-data/top-gainers-loosers',  verify=False, headers=headers)
tree = html.fromstring(resp.content)

count = 1
stocks = []
for i in range(30):
    name = tree.xpath('//*[@id="topgainer-Table"]/tbody/tr['+str(count)+']/td[1]/a')
    print(name)
    try:
        stocks.append(name[0].text)
    except:
        pass
    #name.text  
    count +=1
    
print(stocks)

As an output I get [ ] printed many times (an empty list). I think that the problem is with NSE because it has many different tables all with same xpaths.

Any ideas?

IF the website fetches the data via javascript, you cant use requests/beautifulsoup/urllib style of scraping. Those tools work *ONLY* when the url is static html .. Use Seleniun or Playwright or similar tools. — rasjani, Sep 22 '22 at 09:53

Bhavya Parikh · Accepted Answer · 2022-09-22T09:59:36.023

1

If you check the website it take some time to load data so from bs4 you will not be able to find data.

Network Tab::

You can find it from Network tab and search for any company name you can get from screenshot attached and find the URL and make call to it and it will return JSON data. so you can extract what so data you want

import requests
import pandas  as pd
headers={"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"}
res=requests.get(r"https://www.nseindia.com/api/live-analysis-variations?index=gainers",headers=headers)
all_data=res.json()['NIFTY']['data']
df=pd.DataFrame(all_data)

Output:

URL for top loosers :: https://www.nseindia.com/api/live-analysis-variations?index=loosers

edited Sep 22 '22 at 09:59

answered Sep 22 '22 at 09:54

Bhavya Parikh

3,304
2
9
19

Thanks for that, and if I only want symbols I can do: all_data=res.json()['NIFTY']['data'][1] correct? – gasper101 Sep 22 '22 at 10:00
it is a list so it will return first data you have to iterate over it and then fetch the symbol – Bhavya Parikh Sep 22 '22 at 10:03
You can get all symbols `[i['symbol'] for i in all_data]` and if you need only one `all_data[0]['symbol']` – Bhavya Parikh Sep 22 '22 at 10:08

score 0 · Answer 2 · answered Apr 29 '23 at 05:53

Another easier alternative, Use the nsefetch() function as documented here https://unofficed.com/nse-python/documentation/nsefetch/

In case you want python-requests method

from nsepython import *
payload= nsefetch('https://www.nseindia.com/market-data/top-gainers-loosers')
print(payload)

In case you want curl method

from nsepythonserver import *
payload= nsefetch('https://www.nseindia.com/market-data/top-gainers-loosers')
print(payload)

Python, requests webscraping - NSE India gives emtpy list

2 Answers2