Correct Xpath for element

Question

I am trying to scrape some data from this page.

I am using requests and lxml in python to do so. Specifically I want the ids of the detected topics.

I wrote the following Xpath for them :

'//detectedTopic//@id'

This returned nothing.

Whereas the following worked without any issues:

'//@id'

The developer tools in Chrome showed that the first Xpath indeed points to the correct nodes.

What's wrong with it then?

can you try 2 examples below? //detectedTopic/@id or //detectedTopic — Dmytro Pastovenskyi, Jan 05 '16 at 13:43

score 2 · Accepted Answer · edited May 23 '17 at 12:31

If you use lxml.html to parse the content, then the HTMLParser makes all the tags lowercase since HTML is case-insensitive:

import requests
url = 'http://wikipedia-miner.cms.waikato.ac.nz/services/wikify?source=At%20around%20the%20size%20of%20a%20domestic%20chicken,%20kiwi%20are%20by%20far%20the%20smallest%20living%20ratites%20and%20lay%20the%20largest%20egg%20in%20relation%20to%20their%20body%20size%20of%20any%20species%20of%20bird%20in%20the%20world'
r = requests.get(url)
content = r.content

import lxml.html as LH
html_root = LH.fromstring(content)
print(LH.tostring(html_root))

yields

...
   <detectedtopics>
      <detectedtopic id="17362" title="Kiwi" weight="0.8601778098224363"></detectedtopic>
      <detectedtopic id="21780446" title="Species" weight="0.6213590253455182"></detectedtopic>
      <detectedtopic id="160220" title="Ratite" weight="0.5533763404831633"></detectedtopic>
      <detectedtopic id="37402" title="Chicken" weight="0.528161911497278"></detectedtopic>
   </detectedtopics>

but if you use lxml.etree to parse the content as XML, then the case is not changed:

import lxml.etree as ET
xml_root = ET.fromstring(content)
print(ET.tostring(xml_root))

yields

...
   <detectedTopics>
      <detectedTopic id="17362" title="Kiwi" weight="0.8601778098224363"/>
      <detectedTopic id="21780446" title="Species" weight="0.6213590253455182"/>
      <detectedTopic id="160220" title="Ratite" weight="0.5533763404831633"/>
      <detectedTopic id="37402" title="Chicken" weight="0.528161911497278"/>
   </detectedTopics>

The content looks like XML not HTML, so you should use:

print(xml_root.xpath('//detectedTopic/@id'))
['17362', '21780446', '160220', '37402']

If content is parsed as HTML, then the XPath would need to be lowercased:

print(html_root.xpath('//detectedtopic/@id'))
['17362', '21780446', '160220', '37402']

Great answer! I didn't know the fact that `lxml.html` makes the tags lowercase as I am accustomed to use `lxml.etree` for xml parsing. — salmanwahed, Jan 05 '16 at 14:35

score 0 · Answer 2 · answered Jan 05 '16 at 13:51

0

You can get the ids by this:

'//detectedTopic/@id'

Also you can get the tag and extract your required attributes. Example:

for tag in tr.xpath('//detectedTopic'):
    print tag.attrib.get('id')
    print tag.attrib.get('title')

answered Jan 05 '16 at 13:51

salmanwahed

9,450
7
32
55

Correct Xpath for element

2 Answers2