0

I have the following code:

import os 
import re
from bs4 import BeautifulSoup
import urllib.request
#from urllib.request import request, urlopen
#from urllib import request
import pandas as pd
import numpy as np
import datetime
import time
import openpyxl


for a in range(0,len(symbols),1):
    #ua = UserAgent()
    url = "https://iborrowdesk.com/report/" + symbols[a]
    """
       test urls:
                 https://iborrowdesk.com/report/wmt
    """

    print("Extracting Values for " + symbols[a] + ".")

    try:
        page = urllib.request.urlopen(url)
        soup = BeautifulSoup(page, "html.parser", from_encoding="iso-8859-1")

        value_label_value = soup.find_all('table', {'class': 'table table-condensed table-hover'})
  1. Nothing comes back when I type print(value_label_value)
  2. What am I doing wrong?
Chicken Sandwich No Pickles
  • 2,953
  • 8
  • 45
  • 92
  • 1
    in _{'class': 'table table-condensed table-hover'}_, there are 3 classes listed, instead of one. Try just _{'class': 'table'}_ or _{'class': 'table-condensed'}_. Maybe it can help. – Renato Aranha Apr 03 '21 at 06:04

1 Answers1

1

Data comes from an API call found in network tab. You can re-create table as follows:

import requests
import pandas as pd

data = requests.get('https://iborrowdesk.com/api/ticker/wmt').json()
df = pd.DataFrame(data['real_time'])
df
QHarr
  • 83,427
  • 12
  • 54
  • 101
  • For the sake of my future learning, how did you find that? Where in particular did you look on the network tab? – Chicken Sandwich No Pickles Apr 03 '21 at 06:24
  • See this: https://stackoverflow.com/questions/61049188/how-do-i-get-this-information-out-of-this-website/61051360#61051360 – QHarr Apr 03 '21 at 06:26
  • It looks like I need to look at the "XHR" type, but I'm still not sure which one I should look at on the left in the "name" column to determine how you came to that answer. It solves my problem, just hoping to understand a bit more. – Chicken Sandwich No Pickles Apr 03 '21 at 06:34
  • 1
    It is indeed the xhr type. I normally use the search box to find a particular string or value or search for API or Json. If you go to my profile page there are more examples listed under point 5 in links I commonly share. – QHarr Apr 03 '21 at 06:52