92

I want to extract only the text from the top-most element of my soup; however soup.text gives the text of all the child elements as well:

I have

import BeautifulSoup
soup=BeautifulSoup.BeautifulSoup('<html>yes<b>no</b></html>')
print soup.text

The output to this is yesno. I want simply 'yes'.

What's the best way of achieving this?

Edit: I also want yes to be output when parsing '<html><b>no</b>yes</html>'.

Dragon
  • 2,017
  • 1
  • 19
  • 35
  • 1
    BeautifulSOAP has been removed. To get text of current element only in bs4, refer to @Horst Miller's answer [here](https://stackoverflow.com/a/31909680/4428377) – aquaman Oct 31 '17 at 13:33
  • 1
    The top answers are still valid without BeautifulSOAP. `.find(text=True, recursive=False)` feels cleaner than the `contents` list comp and type check. – ggorlen Jul 20 '21 at 22:37
  • Note that `text` argument to `find` functions is now deprecated. See [my answer below](https://stackoverflow.com/a/76494168/1719931) – robertspierre Jun 17 '23 at 00:00

5 Answers5

90

what about .find(text=True)? 

>>> BeautifulSoup.BeautifulSOAP('<html>yes<b>no</b></html>').find(text=True)
u'yes'
>>> BeautifulSoup.BeautifulSOAP('<html><b>no</b>yes</html>').find(text=True)
u'no'

EDIT:

I think that I've understood what you want now. Try this:

>>> BeautifulSoup.BeautifulSOAP('<html><b>no</b>yes</html>').html.find(text=True, recursive=False)
u'yes'
>>> BeautifulSoup.BeautifulSOAP('<html>yes<b>no</b></html>').html.find(text=True, recursive=False)
u'yes'
jbochi
  • 28,816
  • 16
  • 73
  • 90
40

You could use contents

>>> print soup.html.contents[0]
yes

or to get all the texts under html, use findAll(text=True, recursive=False)

>>> soup = BeautifulSoup.BeautifulSOAP('<html>x<b>no</b>yes</html>')
>>> soup.html.findAll(text=True, recursive=False) 
[u'x', u'yes']

above joined to form a single string

>>> ''.join(soup.html.findAll(text=True, recursive=False)) 
u'xyes'
MendelG
  • 14,885
  • 4
  • 25
  • 52
TigrisC
  • 1,320
  • 9
  • 11
  • 1
    Kinda works, but sadly doesn't help when the html is reversed: `noyes`. I suppose I could iterate over the contents trying to find pieces that aren't tags. – Dragon Feb 14 '11 at 17:40
  • 2
    `findAll(text=True, recursive=False)` thats my jam baby, especially the `recursive=False` bit! – deweydb Apr 17 '21 at 00:57
16

This works for me in bs4:

import bs4
node = bs4.BeautifulSoup('<html><div>A<span>B</span>C</div></html>').find('div')
print "".join([t for t in node.contents if type(t)==bs4.element.NavigableString])

output:

AC
Horst Miller
  • 177
  • 1
  • 2
  • more pythonically isinstance(t, bs4.element.NavigableString) – cards Aug 18 '21 at 22:30
  • nice! be aware that your solution is more general than "find_all(text=True, recursive=False)" since it catch also text around a tag 'hiJso long' you get 'hilong'. Instead with find_all just get the 1st: 'hi' – cards Aug 18 '21 at 22:48
4

You might want to look into lxml's soupparser module, which has support for XPath:

>>> from lxml.html.soupparser import fromstring
>>> s1 = '<html>yes<b>no</b></html>'
>>> s2 = '<html><b>no</b>yes</html>'
>>> soup1 = fromstring(s1)
>>> soup2 = fromstring(s2)
>>> soup1.xpath("text()")
['yes']
>>> soup2.xpath("text()")
['yes']
mzjn
  • 48,958
  • 13
  • 128
  • 248
3

In modern (as of 2023-06-17) BeautifulSoup4, given:

from bs4 import BeautifulSoup
node = BeautifulSoup("""
<html>
    <div>
        <span>A</span>
        B
        <span>C</span>
        D
    </div>
</html>""").find('div')

Use the following to get direct children text elements (BD):

s = "".join(node.find_all(string=True, recursive=False))

And the following to get all descendants text elements (ABCD):

s = "".join(node.find_all(string=True, recursive=True))
robertspierre
  • 3,218
  • 2
  • 31
  • 46