Get the inner HTML of a element in lxml

Question

I am trying to get the HTML content of child node with lxml and xpath in Python. As shown in code below, I want to find the html content of the each of product nodes. Does it have any methods like product.html?

productGrids = tree.xpath("//div[@class='name']/parent::*")
for product in productGrids:
    print #html content of product

score 52 · Accepted Answer · edited Feb 15 '13 at 14:10

52

from lxml import etree
print(etree.tostring(root, pretty_print=True))

you may see more examples here: http://lxml.de/tutorial.html

edited Feb 15 '13 at 14:10

root

76,608
25
108
120

answered Feb 15 '13 at 14:05

Walty Yeung

3,396
32
33

2

Remember it has a optional parameter `encode` use like this: `etree.tostring(xxxxx, encoding='unicode')` – shellbye Nov 17 '16 at 09:32
20

That's *outer* HTML. – Kirill Bulygin Jul 21 '17 at 09:15
1

I juts get `TypeError: Type 'list' cannot be serialized.` – Thomas Weller Feb 10 '21 at 21:55

score 17 · Answer 2 · answered Feb 15 '13 at 14:08

I believe you want to use the tostring() method:

from lxml import etree

tree = etree.fromstring('<html><head><title>foo</title></head><body><div class="name"><p>foo</p></div><div class="name"><ul><li>bar</li></ul></div></body></html>')
for elem in tree.xpath("//div[@class='name']"):
     # pretty_print ensures that it is nicely formatted.
     print etree.tostring(elem, pretty_print=True)

Siva Kannan · Answer 3 · 2019-12-23T07:36:45.983

Simple function to get innerHTML or innerXML
.
Try it out directly https://pyfiddle.io/fiddle/631aa049-2785-4c58-bf82-eff4e2f8bedb/
.

function


def innerXML(elem):
    elemName = elem.xpath('name(/*)')
    resultStr = ''
    for e in elem.xpath('/'+ elemName + '/node()'):
        if(isinstance(e, str) ):
            resultStr = resultStr + ''
        else:
            resultStr = resultStr + etree.tostring(e, encoding='unicode')

    return resultStr

invocation

XMLElem = etree.fromstring("<div>I am<name>Jhon <last.name> Corner</last.name></name>.I work as <job>software engineer</job><end meta='bio' />.</div>")
print(innerXML(XMLElem))

.
Logic Behind

get the outermost element name first,
Then get all the child nodes
Convert all the child nodes to string using tostring
Concatinate Them

score -1 · Answer 4 · answered Jun 07 '17 at 18:26

After right clicking (copy, copy xpath) on the specific field you want (in chrome's inspector), you might get something like this:

//*[@id="specialID"]/div[12]/div[2]/h4/text()[1]

If you wanted that text element for each "specialID"

//*[@id="specialID"]/div/div[2]/h4/text()[1]

You could select another field and it'll interleave the results

//*[@id="specialID"]/div/div[2]/h4/text()[1] | //*[@id="specialID"]/div/some/weird/path[95]

Example could be improved, but it illustrates the point:

//*[@id="mw-content-text"]/div/ul[1]/li[11]/text()

from lxml import html
import requests
page = requests.get('https://en.wikipedia.org/wiki/Web_scraping')
tree = html.fromstring(page.content)
data = tree.xpath('//*[@id="mw-content-text"]/div/ul[1]/li/a/text() | //*[@id="mw-content-text"]/div/ul[1]/li/text()[1]')
print(len(data))
for i in range(len(data)):
    print(data[i])

score -1 · Answer 5 · edited Oct 09 '21 at 21:04

-1

You can use product.text_content() if what you are looking for is the text content rather than the HTML.

edited Oct 09 '21 at 21:04

Sasha Pachev

5,162
3
20
20

answered Mar 13 '19 at 06:24

Virako

650
10
18

1

based on lxml [documentation](https://lxml.de/lxmlhtml.html) `.text_content()` method returns the text content of the element and **not** the inner HTML – Panagiotis Simakis Nov 23 '19 at 12:26
While the answer was not technically correct, it solved my problem, because that is what I was actually looking for. – Sasha Pachev Oct 09 '21 at 21:03

Saurabh Chandra Patel · Answer 6 · 2015-08-20T14:22:25.987

-2

another way to do this

x=doc.xpath("//div[@class='name']/parent::*")
print(map(etree.tostring,x))

edited Aug 20 '15 at 14:22

answered Aug 20 '15 at 14:14

Saurabh Chandra Patel

12,712
6
88
78

Get the inner HTML of a element in lxml

6 Answers6

Linked