Scraping the spot rate from the nse website

Question

I'm new to web scraping and wanted to scrape the Option Chain (Currency Derivatives) table from NSE. I found a similar code in this question: Webscraping NSE Options prices using Python BeautifulSoup, regarding encoding correction but it doesn't extract the value (69.4598) above the table. I want this value that is highlighted in the image but I'm not sure how to extract that and code it in python. I've attached a screenshot of the webpage along with the google inspect element window.

import requests
import pandas as pd
from bs4 import BeautifulSoup

Base_url = ("https://www.nseindia.com/live_market/dynaContent/live_watch/fxTracker/optChainDataByExpDates.jsp")

page = requests.get(Base_url)

soup = BeautifulSoup(page.content, 'html.parser')
table_it = soup.find_all(class_="opttbldata")
spot = soup.find(id = "td")
table_cls_1 = soup.find_all(id = "octable")
col_list = []

for mytable in table_cls_1:
    table_head = mytable.find('thead')

    try:
        rows = table_head.find_all('tr')
        for tr in rows:
            cols = tr.find_all('th')
            for th in cols:
                er = th.text
                ee = er.encode('utf8')
                col_list.append(ee)
    except:
        print('no thread')
        
col_list_fnl = [e for e in col_list if e not in ('CALLS', 'PUTS', 'Chart', '\xc2\xa0')]

table_cls_2 = soup.find(id = "octable")
all_trs = table_cls_2.find_all('tr')
req_row = table_cls_2.find_all('tr')

new_table = pd.DataFrame(index=range(0,len(req_row)-3),columns = col_list_fnl)

row_marker = 0

for row_number, tr_nos in enumerate(req_row):
    if row_number <= 1 or row_number == len(req_row)-1:
        continue # To insure we only choose non empty rows

    td_columns = tr_nos.find_all('td')

    # Removing the graph column
    select_cols = td_columns[1:22]
    cols_horizontal = range(0,len(select_cols))

    for nu, column in enumerate(select_cols):

        utf_string = column.get_text()
        utf_string = utf_string.strip('\n\r\t": ')
        tr = utf_string.encode('utf8')

        new_table.iloc[row_marker,[nu]] = tr

    row_marker += 1

print(new_table)

score 2 · Accepted Answer · answered Jun 12 '19 at 09:52

You can grab all the <td> tags and iterate through those to see where is contains the substring 'REFERENCE RATE - FBIL'. It actually shows up 4 times, so put it into a list, then just print out the element.

import requests
import pandas as pd
from bs4 import BeautifulSoup

Base_url = ("https://www.nseindia.com/live_market/dynaContent/live_watch/fxTracker/optChainDataByExpDates.jsp")

page = requests.get(Base_url)

soup = BeautifulSoup(page.text, 'html.parser')
tds = soup.find_all('td')

refRates = [ each.find('strong').text for each in tds if 'REFERENCE RATE - FBIL' in each.text ]
print (refRates[0])

Output:

print (refRates[0])
69.4598

@DevanshiRuparel be sure to accept the solution if it worked out — chitown88, Jun 12 '19 at 12:03

score 1 · Answer 2 · answered Jun 12 '19 at 10:11

As BeautifulSoup has the capability to play with pseudo selectors you can always take this advantage to make your script robust. Check out the implementation below:

import requests
from bs4 import BeautifulSoup

url = "https://www.nseindia.com/live_market/dynaContent/live_watch/fxTracker/optChainDataByExpDates.jsp"

page = requests.get(url)
soup = BeautifulSoup(page.text,'html.parser')
item = soup.select_one("div:contains('REFERENCE RATE') > strong").text
print(item)

Output at this moment:

69.4222

Scraping the spot rate from the nse website

2 Answers2