Scrapy scraping nested text using css selectors

Question

I have the following html code:

<div class='article'>
<p>Lorem <strong>ipsum</strong> si ammet</p>
</div>

So to get the text data as: Lorem ipsum si ammet, so I tried to use:

response.css('div.article >p::text ').extract()

But I only receive only lorem sie ammet.

How can I get both <p> and <strong> texts using CSS selectors?

Doesn't look like a duplicate to me. This question asks for a way specifically using CSS selectors, while the other one only mentions XPath selectors. — Attila, Apr 25 '18 at 22:51

score 4 · Accepted Answer · answered Mar 27 '18 at 15:21

One liner solution.

"".join(a.strip() for a in response.css("div.article *::text").extract())

div.article * means to scrape everything inside the div.article

Or an easy way to write it

text = ""
for a in response.css("div.article *::text").extract()
    text += a.strip()

Both approaches are same,

score 0 · Answer 2 · answered Nov 14 '22 at 10:58

0

In Scrapy 2.7+ you can do so with following

text = response.css('div.article *::text').getall()
text = [t.strip() for t in text]
text = "".join(text)

getall() method returns list.

answered Nov 14 '22 at 10:58

i-naeem

2 Answers2