How to set the mechanize page encoding?

Question

I'm trying to get a page with an ISO-8859-1 encoding clicking on a link, so the code is similar to this:

page_result = page.link_with( :text => 'link_text' ).click

So far I get the result with a wrong encoding, so I see characters like:

'T�tulo:' instead of 'Título:'

I've tried several approaches, including:

Stating the encoding in the first request using the agent like:

@page_search = @agent.get(
  :url => 'http://www.server.com',
  :headers => { 'Accept-Charset' => 'ISO-8859-1' } )

Stating the encoding for the page itself
```
  page_result.encoding = 'ISO-8859-1'
```

But I must be doing something wrong: a simple puts always show the wrong characters.

Do you know how to state the encoding?

Thanks in advance,

Added: Executable example:

require 'rubygems'
require 'mechanize'

WWW::Mechanize::Util::CODE_DIC[:SJIS] = "ISO-8859-1"

@agent = WWW::Mechanize.new

@page = @agent.get(
  :url => 'http://www.mcu.es/webISBN/tituloSimpleFilter.do?cache=init&layout=busquedaisbn&language=es',
  :headers => { 'Accept-Charset' => 'utf-8' } )

puts @page.body

score 10 · Answer 1 · answered Nov 25 '11 at 11:26

10

Hey you can just do a:

agent.page.encoding = 'utf-8'

Hope it helps!

answered Nov 25 '11 at 11:26

Niels Kristian

8,661
11
59
117

It hard to use 'agent.page.encoding' in real code, but idea is correct and very helpful, thanks! – denis.peplin Apr 25 '12 at 10:40
I spent many hours trying to fix this problem until I stumbled upon your answer -- thank you! – CodeBiker Jul 09 '13 at 21:01

score 4 · Answer 2 · answered Apr 25 '12 at 10:39

4

The previous answer is correct, but in my code it looks slightly different:

agent = Mechanize.new

page = agent.get('http://example.com')

page.encoding = 'windows-1251'

page.search('p').each do |para|
  puts para.text
end

answered Apr 25 '12 at 10:39

denis.peplin

9,585
3
48
55

Thanks! This tip fixed a related issue regarding form encoding that was in ASCII. – Joe Chen Apr 16 '13 at 18:24

score 1 · Answer 3 · answered Dec 14 '09 at 03:07

1

Sorry, it was my mistake: I come from a Java background and there strings are internally converted to utf-16. I forgot Ruby doesn't do it. Mechanize was recovering the page flawlessly, but I needed to convert the data via iconv.

Mental note: Ruby stores the strings without converting its encoding.

answered Dec 14 '09 at 03:07

Juan

989
1
14
22

you might also wanna try ruby 1.9 if possible, they added a whole lot of [unicode stuff](http://blog.nuclearsquid.com/writings/ruby-1-9-encodings) – Marc Seeger Dec 15 '09 at 08:36

score 0 · Answer 4 · edited May 23 '17 at 11:59

0

Yeah, Mechanize will try to detect the encoding itself (using the NKF core Ruby library) to guess the encoding) and sometimes fails.

Maybe this might help:
WWW::Mechanize::Util::CODE_DIC[:SJIS] = "ISO-8859-1"

I'm not too sure about the exact syntax, but I think the CODE_DICT Hash might be a good place to look :)
I had a similar problem a while back.

edited May 23 '17 at 11:59

Community

1
1

answered Dec 12 '09 at 11:21

Marc Seeger

2,717
4
28
32

How to set the mechanize page encoding?

4 Answers4

Linked