Character encoding with Ruby 1.9.3 and the mail gem

Question

I'm trying to parse email strings with the Ruby mail gem, and I'm having a devil of a time with character encodings. Take the following email:

MIME-Version: 1.0
Sender: foobar@example.com
Date: Thu, 14 Jun 2012 09:00:18 -0400
Message-ID: <CAGErOzF3FV5NvzN3zUpLGPok96SFzK18Z4HerzyYNALnzgMVaA@mail.gmail.com>
Subject: Re: [Lorem Ipsum] Foo updated the forum topic 'Reply by email test'
From: Foo Bar <foo@example.com>
To: Foo <c49964d167e08e7d4a1930e6565f23c258be19a0@foo.example.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

This email has accents:=A0R=E9sum=E9
>
> --------- Reply Above This Line ------------
>
> Email parsing with accents: R=E9sum=E9
>
> Click here to view this post in your browser

The email body, when properly encoded, should be:

This reply has accents: Résumé
>
> --------- Reply Above This Line ------------
>
> Email parsing with accents: Résumé
>
> Click here to view this post in your browser

However, I'm having a devil of a time actually getting the accent marks to come through. Here's what I've tried:

message = Mail.new(email_string)
body = message.body.decoded

That gets me a string that starts like this:

This reply has accents:\xA0R\xE9sum\xE9\r\n>\r\n> --------- Reply Above This Line ------------

Finally, I try this:

body.encoding # => <Encoding:ASCII-8BIT>
body.encode("UTF-8") # => Encoding::UndefinedConversionError: "\xA0" from ASCII-8BIT to UTF-8

Does anyone have any suggestions on how to deal with this? I'm pretty sure it has to do with the "charset=ISO-8859-1" setting in the email, but I'm not sure how to use that, or if there's a way to easily extract that using the mail gem.

score 18 · Accepted Answer · edited Mar 17 '15 at 16:38

18

After playing a bit, I found this:

body.decoded.force_encoding("ISO-8859-1").encode("UTF-8") # => "This reply has accents: Résumé..."
message.parts.map { |part| part.decoded.force_encoding("ISO-8859-1").encode(part.charset) } # multi-part

You can extract the charset from the message like so.

message.charset #=> for simple, non-multipart
message.parts.map { |part| part.charset } #=> for multipart, each part can have its own charset

Be careful with non-multipart, as the following can cause trouble:

body.charset #=> returns "US-ASCII" which is WRONG!
body.force_encoding(body.charset).encode("UTF-8") #=> Conversion error...

body.force_encoding(message.charset).encode("UTF-8") #=> Correct conversion :)

edited Mar 17 '15 at 16:38

Jake Berger

5,237
1
28
22

answered Jun 14 '12 at 18:54

Micah

17,584
8
40
46

1

Awesome. Been looking for this. Ended up doing this: body = message.text_part.encode('UTF-8', message.text_part.charset, :invalid => :replace, :undef => :replace) – Kieran Pilkington Feb 26 '13 at 21:22
Some parts seem to have a nil charset. I'm not sure yet how to handle those. – Thibaut Barrère Feb 10 '14 at 14:59
3

Good to know! Does anybody know why the mail gem isn't doing that automatically? – moritz Jun 26 '14 at 08:40

score 0 · Answer 2 · edited May 23 '17 at 12:12

0

This didn't work for me, so thought I'd stick up the solution I got to in case it helps anyone...

Basically had to add encoding defaults and tweak the output into sensible strings. https://stackoverflow.com/a/26604049/2386548

edited May 23 '17 at 12:12

Community

1
1

answered Oct 28 '14 at 08:36

Carpela

2,155
1
24
55

Character encoding with Ruby 1.9.3 and the mail gem

2 Answers2