Having trouble using Beautiful Soup's 'Next Sibling' to extract some information

Question

On Auction websites, there is a clock counting down the time remaining. I am trying to extract that piece of information (among others) to print to a csv file.

For example, I am trying to take the value after 'Time Left:' on this site: https://auctionofchampions.com/Michael_Jordan___Magic_Johnson_Signed_Lmt__Ed__Pho-LOT271177.aspx

I have tried 3 different options, without any success

1)

time = ''
    try:
        time = soup.find(id='tzcd').text.replace('Time Left:','')
        #print("Time: ",time)
    except Exception as e:
        print(e)

    time = ''
    try:
        time = soup.find(id='tzcd').text
        #print("Time: ",time)
    except:
        pass

3

time = ''
  try:
      time = soup.find('div', id="BiddingTimeSection").find_next_sibling("div").text
      #print("Time: ",time)
  except:
      pass

I am a new user of Python and don't know if it's because of the date/time structure of the pull or because of something else inherently flawed with my code.

Any help would be greatly appreciated!

you may have the most common problem: page may use `JavaScript` to add/update elements but `BeautifulSoup`/`lxml`, `requests`/`urllib` can't run `JS`. You may need [Selenium](https://selenium-python.readthedocs.io/) to control real web browser which can run `JS`. OR use (manually) `DevTools` in `Firefox`/`Chrome` (tab `Network`) to see if `JavaScript` reads data from some URL. And try to use this URL with `requests`. `JS` usually gets `JSON` which can be easy converted to Python dictionary (without `BS`). You can also check if page has (free) `API` for programmers. — furas, Aug 07 '22 at 10:08

score 1 · Answer 1 · answered Aug 06 '22 at 22:56

That information is being pulled into page via a Javascript XHR call. You can see that by inspecting Network tab in browser's Dev tools. The following code will get you the time left in seconds:

import requests

s = requests.Session()

header = {'X-AjaxPro-Method': 'GetTimerText'}
payload = '{"inventoryId":271177}'
r = s.get('https://auctionofchampions.com/Michael_Jordan___Magic_Johnson_Signed_Lmt__Ed__Pho-LOT271177.aspx')
s.headers.update(header)
r = s.post('https://auctionofchampions.com/ajaxpro/LotDetail,App_Web_lotdetail.aspx.cdcab7d2.1voto_yr.ashx', data=payload)
print(r.json()['value']['timeLeft'])

Response:

792309 seconds are a bit over 9 days. There are easy ways to return them in days/hours/minutes, if you want.

Having trouble using Beautiful Soup's 'Next Sibling' to extract some information

1 Answers1