1

Our production rails server receives a post request (hook) from an external service (sparkpost) with the following format

data: {
...
"subject": "Your RedvanlyCategory: Men’s,<br>Redvanly Kent Pant, XL in Estate Blue arrived!",
...
}

Mind the the apostrophe character ’ not '. This breaks the rails request stack at:

[4ce93af4ed8b] [28a6b892-2c45-409b-90cf-3d1b4fa9b5f7] no implicit conversion of nil into String excluded from capture: DSN not set
[4ce93af4ed8b] [28a6b892-2c45-409b-90cf-3d1b4fa9b5f7]   
[4ce93af4ed8b] [28a6b892-2c45-409b-90cf-3d1b4fa9b5f7] ActionDispatch::Http::Parameters::ParseError (no implicit conversion of nil into String):
[4ce93af4ed8b] [28a6b892-2c45-409b-90cf-3d1b4fa9b5f7]   
[4ce93af4ed8b] [28a6b892-2c45-409b-90cf-3d1b4fa9b5f7] actionpack (5.1.7) lib/action_dispatch/http/parameters.rb:115:in `rescue in parse_formatted_parameters'

Why does this happen? The apostrophe seems to be a valid unicode character. Changing from ’ to ' no longer breaks the rails stack.

The header of the request is :

Accept  application/json
Accept-Encoding gzip
Content-Length  3971
Content-Type    application/json
Host    4ce93af4ed8b.ngrok.io
User-Agent  SparkPost
X-Forwarded-For 52.37.3.48
X-Forwarded-Proto   http

Edit: The curl to reproduce

curl --location --request GET 'http://localhost:3000/receive_sparkpost_hooks' \
--header 'Content-Type: application/json' \
--data-raw '[{"subject":"Your RedvanlyCategory: Men’s,<br>Redvanly Kent Pant, XL in Estate Blue arrived!"}]'

Previously we've seen some unicode characters (sparkpost & JSON should support UTF-8) that we're were crashing the rails stack and we've filtered them out using

encode('ASCII', 'binary', invalid: :replace, undef: :replace, replace: '')

The characters looked like enter image description here

I may believe we're dealing with this the wrong way and could use some advice in how to feed data to the service which in turn won't feed the rails API badly formatted unicode text.

Ion Andrei Bara
  • 74
  • 8
  • 1
    Is it a backtick or an apostrophe? – Rockwell Rice Jun 24 '21 at 14:29
  • does not seem to be a backtick (`) but ’ (it's the same character as you see it in this post) – Ion Andrei Bara Jun 24 '21 at 14:42
  • 1
    What lead you to believe it was the apostrophe, I don't see it pointing there in the stacktrace you posted – Int'l Man Of Coding Mystery Jun 24 '21 at 15:32
  • This is a "Right Apostrophe" or "Right Single Quotation Mark" (https://www.compart.com/en/unicode/U+2019) – engineersmnky Jun 24 '21 at 15:32
  • sounds like rails is not happy with the encoding of the request – dbugger Jun 24 '21 at 16:06
  • I'd look at the routine that is bringing this in. assuming that the POST isn't failing at the POST stage, and more at the time that the code is being handled. My guess is that something is wrapping the submit in single quotes, and that your embedded single-quote is breaking the string early – Jad Jun 25 '21 at 10:54
  • @Int'lManOfCodingMystery removing it no longer raises an exception in the rails libraries. i have updated the full culprit key value – Ion Andrei Bara Jun 25 '21 at 15:41
  • @engineersmnky seems to match the "Right Single Quotation Mark" as in the link you have provided (pasted the character next to the original in the DOM and they look pretty much alike) – Ion Andrei Bara Jun 25 '21 at 15:43
  • @IonAndreiBara can you post more code as to where this is occurring? It looks like this should be using the json parser but I cannot recreate the issue when I parse this as JSON. – engineersmnky Jun 25 '21 at 16:26
  • @engineersmnky i have updated the question text with the curl call which reproduces the issue on my localhost. i'd mention that we've seen some 'badly encoded unicode characters' we're pushing to the service that was filtered out using .encode('ASCII', 'binary', invalid: :replace, undef: :replace, replace: '') - added this section in the question. – Ion Andrei Bara Jun 26 '21 at 18:32

1 Answers1

0

The cause was a non-break unicode character present in the string (https://unicode-table.com/en/00A0/) yet not removed through the applied filtering in the question's text.

Ion Andrei Bara
  • 74
  • 8
  • After 2 years: Turns out any non ascii character in a json crashes rails 5 Example: POST { "a" : "asÃdfs" } results in parserError exception before hitting any user code – Ion Andrei Bara Apr 03 '23 at 12:39