IndexError by using Xpath for scraping

Question

I created this code in order to get real time amazon index market data from the platform PLUS 500:


import requests
import bs4
from lxml import etree
from Config import username, password

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Connection": "keep-alive"
}

with requests.session() as session:
    url = 'https://app.plus500.com/trade/amazon'
    page = requests.get(url, auth=(username, password), headers=headers)
    try:
        page.raise_for_status()
    except Exception as exc:
        print(f'Problem: {exc}')
    html_page = bs4.BeautifulSoup(page.content, 'html.parser')
    print(html_page.prettify())
    dom = etree.HTML(str(html_page))
    print(dom.xpath('//*[@id="_win_plus500_bind818"]')[0].text)

I created a dummy account for you in case it is necessary Username: myrandomcode@gmail.com Password: MyRandomCode87

the problem I get back is this one


Traceback (most recent call last):
  File "############", line 24, in <module>
    print(dom.xpath('//*[@id="_win_plus500_bind818"]')[0].text)
IndexError: list index out of range

I am trying to scrape the sell and buy prices.

Hello, Emasei, I hope you were able to solve the problem with the answer I provided. If so, I would be very grateful if you could mark it as 'accepted', thank you. :) — Pexers, Sep 04 '22 at 01:05

score 1 · Answer 1 · answered Aug 28 '22 at 13:06

1

The error you're getting is related to the indexing operation [0].text.

Make sure the element you're trying to access exists. Also, make sure the @id you're using doesn't change when you refresh the web page ("_win_plus500_bind818").

answered Aug 28 '22 at 13:06

Pexers

953
1
7
20

score 0 · Answer 2 · answered Aug 28 '22 at 13:11

The page you are trying to parse is dynamic, meaning the content is loaded after loading the page itself. To parse the dynamic page, you have to use selenium, for example.

More details you can find in the following answer: How can I parse a dynamic page using Python?

IndexError by using Xpath for scraping

2 Answers2