2

Update: After further investigation I've managed to narrow the problem down to the json encoder. Passing the input straight through works fine, but passing it through MultiJson.encode is what's causing the problem.

I'm sending the following up to a restful web service via curl:

$ curl -v -X POST "http://my/url" -d "{\"body\": \"\"}"

The character that you probably can't see is the Credit Card emoji character, which is U+1F4B3.

The response I get back from the service is essentially:

< HTTP/1.1 200 OK
< Date: Wed, 30 Oct 2013 02:38:04 GMT
< Content-Type: application/json;charset=utf-8
< Content-Length: 266
< Connection: close
< 
{ [data not shown]
100   304  100   266  100    38    936    133 --:--:-- --:--:-- --:--:--   936
* Closing connection 0
{
  "body": "\uf4b3"
}

This encoded character does not correspond to what I sent and I would expect it to be returned as sent (in this case).

I have access to the server's source code. It's built on Ruby, Sinatra and ActiveRecord. There is some amount of processing going on before the response is sent:

  • First the content is passed through ERB::Util.html_escape
  • Then, a series of regexs are applied via str.gsub!(reg, " ### ")
  • Finally, the response is returned via MultiJson.encode

I'm not a Ruby person, but can provide additional details if necessary. Would appreciate someone pointing me in the right direction. Thanks!

twaddington
  • 11,607
  • 5
  • 35
  • 46
  • 1
    Something goes wrong in the encoding of U+1F4B3. Using the `\u` notation, it should appear as two units (a surrogate pair), but apparently the software that converts it is not prepared to handle non-BMP characters (characters > U+FFFF) and erroneously just masks 1F4B3 to F483. – Jukka K. Korpela Oct 30 '13 at 05:14
  • Something you didn't tell us: what does your "server" do? :) Does it echo what you send it? – John Bachir Oct 30 '13 at 05:19
  • @JohnBachir it creates an object that sanitizes the data and stores it in a database. It returns the created object serialized to JSON. – twaddington Oct 30 '13 at 05:31

2 Answers2

0

The first thing to check is if the character is getting "into" the body of your application the way you think it does. Ruby has a notion of default "internal" and "external" encoding. And once a string gets in via various IO, for various reasons it may or may not have the expected encoding as it's passed around.

Which isn't to say that it's hard to manage or confusing — it's all pretty straightforward, but I'm just pointing out that all these things can possibly be configured/changed.

To see what you are starting with, as soon as you can in your program, once you have the input, check its encoding.

params[:foo].encoding
=> #<Encoding:UTF-8>

If it's not utf-8 then you need to set your environment and/or your IO mechanism to use utf-8.

Starting in ruby 2.0, the default encoding is — praise the gods — utf8. So if you aren't using ruby 2.0 and are able to, start with upgrading to that.

If you don't have that option, then you need to set the default encoding. Although it seems sinatra sets it to utf-8.

John Bachir
  • 22,495
  • 29
  • 154
  • 227
0

We were able to solve this problem by migrating to a different JSON encoding engine:

get "/foo" do
    resp = ""

    puts MultiJson.adapter()
    puts MultiJson.dump(resp) # Fails

    MultiJson.engine = :jrjackson
    puts MultiJson.adapter()
    puts MultiJson.dump(resp) # Succeeds
end
twaddington
  • 11,607
  • 5
  • 35
  • 46