3

How do I extract <content:encoded> ... </content:encoded> content using scrapy XPath from an RSS feed (sample below)?

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
     xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Latest &#8211; Reason.com</title>
    <item>
        <pubDate>Thu, 16 Jan 2020 21:40:23 +0000</pubDate>
        <content:encoded><![CDATA[<p><span style="font-weight: 400">
          Jimmy Meders was scheduled to die by lethal injection today, 
          but the Georgia parole board has granted him clemency.</span></p>]]> 
        </content:encoded>
...

I tried response.xpath('//content:encoded').get(), but it's not working.

Any help is highly appreciated.

kjhughes
  • 106,133
  • 27
  • 181
  • 240
ashwin
  • 332
  • 2
  • 16

1 Answers1

7

You have to declare and register an XML namespace prefix:

response.selector.register_namespace('content', 
                                     'http://purl.org/rss/1.0/modules/content/')
response.xpath('//content:encoded').getall()

Documentation: register_namespace()

kjhughes
  • 106,133
  • 27
  • 181
  • 240