Detect encoding

Question

I'm getting some string data from the web, and I suspect that it's not always what it says it is. I don't know where the problem is, and I just don't care any more. From day one on this project I've been fighting Ruby string encoding. I really want some way to say: "Here's a string. What is it?", and then use that data to get it to UTF-8 so that it doesn't explode gsub() 2,000 lines down in the depths of my app. I've checked out rchardet, but even though it supposedly works for 1.9 now, it just blows up given any input with multiple bytes... which is not helpful.

In Ruby, you can inspect a string's encoding with `str.encode.name`. You can specify the encoding of a stream / file at the time of connection / opening. E.g. `fh = File.open(file_in,"rb:utf-16le")` (notice, non-8 bit encodings need a binary read). When writing to a file, you should also specify the encoding. — Assad Ebrahim, Nov 10 '15 at 11:21

score 10 · Answer 1 · answered Apr 18 '16 at 12:44

You can't really detect the encoding. You can only assume it.

For the most Western languages applications, the following construct will work. The traditional encoding usually is "ISO-8859-1". The new and preferred encoding is UTF-8. Why not simply try to encode it with UTF-8 and fallback with the old encoding

def detect_encoding( str )
  begin
    str.encode("UTF-8")
    "UTF-8"
  rescue
    "ISO-8859-1"
  end
end

score 8 · Answer 2 · answered Jun 19 '10 at 11:20

8

It is impossible to tell from a string what encoding it is in. You always need some additional metadata that tells you what the string's encoding is.

If you get the string from the web, that metadata is in the HTTP headers. If the HTTP headers are wrong, there is absolutely nothing that you or Ruby or anyone else can do. You need to file a bug with the webmaster of the site where you got the string from and wait till he fixes it. If you have a Service Level Agreement with the website, file a bug, wait a week, then sue them.

answered Jun 19 '10 at 11:20

Jörg W Mittag

363,080
75
446
653

8

It is impossible to know for certain, but you can make an educated guess. That is what rchardet does. However, I haven't been able to get it to work with Ruby 1.9. – ThomasW Nov 08 '11 at 08:39
1

You can definitely make a solid guess. Given a file of decent length, if you see a bunch of valid UTF8 sequences and no invalid UTF8 sequences, you can detect UTF8. If no high bytes, you can guess ASCII. Depending where the program is deployed, you can guess one of the western european encodings - 8859-15 or Windows-1252 most commonly - but you're very much into shaky territory here. The point is, it's a sliding scale of context-dependent probability, not a binary impossibility. – Barry Kelly Mar 20 '18 at 17:56

score 3 · Answer 3 · answered Jun 12 '17 at 12:16

why not try use https://github.com/brianmario/charlock_holmes to get the exact encoding. Then also use it to convert to UTF8

    require 'charlock_holmes'
    class EncodeParser
      def initialize(text)
        @text = text
      end

      def detected_encoding
        CharlockHolmes::EncodingDetector.detect(@text)[:encoding]
      end

      def convert_to_utf8
        CharlockHolmes::Converter.convert(@text, detected_encoding, "UTF-8")
      end
    end

then just use EncodeParser.new(text).detected_encoding or EncodeParser.new(text). convert_to_utf8

score 3 · Answer 4 · answered Mar 12 '12 at 17:15

3

Old question, but chardet works on 1.9: http://rubygems.org/gems/chardet

answered Mar 12 '12 at 17:15

Carson Reinke

713
5
16

1

Never mind, looks like I am wrong about this. String#encoding on 1.9 tells you what encoding the string is in. – Carson Reinke Mar 14 '12 at 17:16

score 2 · Answer 5 · answered Jul 28 '12 at 10:03

We had some fine experience with ensure_encoding. It actually does the job for us to convert resource files having unknown encoding to UTF-8.

The README will give you some hints which options would be a good fit for your situation.

I have never tried chardet since ensure_encoding did the job just fine for us.

I covered here how we use ensure_encoding.

score 1 · Answer 6 · edited Sep 01 '11 at 15:02

1

Try setting these in your environment.

export LC_ALL=en_US.UTF-8
export LC_CTYPE=en_US.UTF-8

Try ruby -EBINARY or ruby -EASCII-8BIT to command line

Try adding -Ku or -Kn to your ruby command line.

Could you paste the error message ?

Also try this: http://github.com/candlerb/string19/blob/master/string19.rb

edited Sep 01 '11 at 15:02

John Bachir

22,495
29
154
227

answered Jun 19 '10 at 11:17

rahul

598
1
9
20

score 0 · Answer 7 · answered Sep 08 '22 at 13:35

0

I know it's an old question, but in modern versions of Ruby it's as simple as str.encoding. You get a return value something like this: #Encoding:UTF-8

answered Sep 08 '22 at 13:35

Jimbo

1,685
3
12
15

This is indeed the solution for modern Ruby. Please vote this up. – naudster Mar 10 '23 at 00:25

score -1 · Answer 8 · answered Jun 19 '10 at 08:11

-1

Might try reading this: http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/

answered Jun 19 '10 at 08:11

thomasfedb

5,990
2
37
65

Detect encoding

8 Answers8