5

I am wondering if there's a way to detect non-ASCII characters in Rails.

I have read that Rails does not use Unicode by default, and characters like Chinese and Japanese have assigned ranges in Unicode. Is there an easy way to detect these characters in Rails? or just specify the range of characters I am expecting?

Is there a plugin for this? Thanks in advance!

edgerunner
  • 14,873
  • 2
  • 57
  • 69
gerky
  • 6,267
  • 11
  • 55
  • 82

2 Answers2

7

All ideographic language encodings use multiple bytes to represent a character, and Ruby 1.9+ is aware of the difference between bytes and characters (Ruby 1.8 isn't)

You may compare the character length to the byte length of the string as a quick and dirty detector. It is probably not foolproof though.

class String
  def multibyte?
    chars.count < bytes.count
  end
end

"可口可樂".multibyte? #=> true
"qwerty".multibyte? #=> false
edgerunner
  • 14,873
  • 2
  • 57
  • 69
  • Thanks! but I used regex to match them, like matching {Han} and others – gerky Sep 16 '11 at 05:13
  • Regex is good and more foolproof and precise than this, but this is probably much faster than regex. – edgerunner Sep 16 '11 at 05:26
  • 1
    This method seems quite clever. Thanks! – Henley Jun 29 '13 at 19:29
  • 1
    Easy smart solution- to clarify, this will distinguish between the 128 US-ASCII characters in Unicode, which need one byte, and everything else- including all foreign alphabets but also things like copyright symbols. (Info here: http://en.wikipedia.org/wiki/UTF-8 and http://en.wikipedia.org/wiki/List_of_Unicode_characters) – Yarin Jul 09 '14 at 16:25
1

This is pretty easy with 1.9.2 as regular expressions are character-based in 1.9.2 and 1.9.2 knows the difference between bytes and characters top to bottom. You're in Rails so you should get everything in UTF-8. Happily, UTF-8 and ASCII overlap for the entire ASCII range so you can just remove everything that isn't between ' ' and '~' when you have UTF-8 encoded text:

>> "Wheré is µ~pancakes ho元use?".gsub(/[^ -~]/, '')
=> "Wher is ~pancakes house?"

There's really no reason to go to all this trouble though. Ruby 1.9 works great with Unicode as does Rails and pretty much everything else. Dealing with non-ASCII text was a nightmare 15 years ago, now it is common and fairly straight forward.


If you do manage to get text data that isn't UTF-8 then you have some options. If the encoding is ASCII-8BIT or BINARY then you can probably get away with s.force_encoding('utf-8'). If you end up with something other than UTF-8 and ASCII-8BIT then you can use Iconv to re-encode it.

References:

mu is too short
  • 426,620
  • 70
  • 833
  • 800
  • Thanks! also, do you know how to test the filtering of foreign characters? in rspec? – gerky Sep 04 '11 at 01:53
  • @mr_lu_kim: The same way you'd test any other string manipulation in RSpec. You'd just do various `utf8_string.mangle.should == utf8less_string` and such. – mu is too short Sep 04 '11 at 02:10