Parsing HTML with a weird encoding with Nokogiri

Question

I can't use XPath because the encoding gets weird. I hoped you could help me out of this trouble.

require "Nokogiri"
require "open-uri"
link = "http://www.arla.dk/Services/SearchService.asmx/RecipeResult?q=allRecipe&paging=6&include=&exclude=&area=recipeSearch&languageBranch=da"
doc = Nokogiri::HTML(open(link))
doc.xpath("//h2")

The xpath method returns an empty array. It looks like the document has not been parsed correct. I think it is due to the file being parsed contains the encoded characters:

&lt;strong&gt;Frokost til 8&lt;/strong&gt;
&lt;ul&gt;&lt;li class='ingHeading'&gt;&lt;strong&gt;&lt;b&gt;Flade

This is not a nokogiri problem, it's a problem with the source data, which is encoded as one long string. — Chris Salzberg, Oct 30 '12 at 09:29
Interesting, I will see if I can find some tool for converting it. Do you know of any? — user1785060, Oct 30 '12 at 09:32

score 1 · Accepted Answer · edited Nov 01 '12 at 18:46

1

The response is XML so first parse it with Nokogiri::XML:

xml = Nokogiri::XML open(link)

then the first string contains some HTML so parse that with Nokogiri::HTML

doc = Nokogiri::HTML xml.at('string').text

Now you can do your search:

doc.xpath '//h2'

edited Nov 01 '12 at 18:46

the Tin Man

158,662
42
215
303

answered Oct 30 '12 at 10:09

pguardiario

53,827
19
119
159

score 0 · Answer 2 · edited May 23 '17 at 12:12

0

As stated above, the issue is that the HTML is encoded, which is why you are seeing escape sequences; For example, < instead of <. To get around it, unescape the HTML.

"How do I encode/decode HTML entities in Ruby? basically suggests using htmlentities.

edited May 23 '17 at 12:12

Community

1
1

answered Oct 30 '12 at 10:00

AJcodez

31,780
20
84
118

Parsing HTML with a weird encoding with Nokogiri

2 Answers2