20

Ok I am running a public JSONP API which the data is served from my PHP server. I just read this article:

Basically if my JSON strings contains a U+2028 character (Unicode line separator) or U+2029 character (Unicode paragraph separator) then this is perfectly valid JSON. However when using JSONP the JSON gets executed as JavaScript and no string in JavaScript can contain a literal U+2028 or a U+2029 as it will break the JavaScript. Apparently this is usually not a problem as long as you use a proper JSON parser, but in the case of JSONP the browser is the JSON parser.

Essentially if these characters were inside strings in my JSONP data being sent to the client this would throw a line or paragraph break into the string which would break the JavaScript and stop it executing. This is a possibility as the API is sending back some client entered data. Someone could potentially enter a U+2028 or a U+2029 into the database, so when I send that back as JSONP it will break any implementation using my API.

So my question is, in PHP how can I sanitise/output escape the JSON data to remove or escape the U+2028 and U+2029 characters before sending it to the client?

Currently my process is doing a json_encode on an array of data and sending that data down to the client. Should I escape the data by looping through the array and filtering it, or escape all the JSON encoded string all at once?

The other thing is I'm not sure how to escape the U+2028 and U+2029 characters in PHP anyway. Can I just do a str_replace? I'm not sure if str_replace is multibyte safe and there's no mb_str_replace function unless I use some custom made one. So how do you remove/escape those unicode characters?

Thanks very much.

hakre
  • 193,403
  • 52
  • 435
  • 836
zuallauz
  • 4,328
  • 11
  • 43
  • 54
  • 1
    Unless I read the article incorrectly, can't you just replace it with `\u2028`? – hafichuk Jan 06 '13 at 04:57
  • `\u2028` and `u2029` are unicode newline characters. Do you want your application to allow these characters in the strings? Meaning, whatever accepts this text on your server actively converts CR/LF into one of these characters, because virtually no one uses the new newline characters. I doubt this. I'm guessing you just want to eliminate all newlines, either CR/LF, or the new ones. In which cases, simply eliminate them, instead of inserting literal escape sequences. – Milind R Dec 11 '14 at 18:13

2 Answers2

23

You can replace U+2028, U+2029 with "\u2028", "\u2029" either on the PHP side or the JavaScript side, or both, it doesn't matter as long as it happens at least once (it's idempotent).

You can just use ordinary string replacement functions. They don't need to be "multibyte safe", and you can do it just as easily in any Unicode encoding (UTF-8, UTF-16, UTF-32 are all equally fine). PHP didn't have Unicode escape sequences last time I checked which is just one more reason why PHP is a joke but you can use the \x escape with UTF-8...

(In short, the reason there's no multibyte string replace function is because it would be redundant -- it would be exactly the same as a non-multibyte string replace function.)

// Javascript
data = data.replace("\u2028", "\\u2028").replace("\u2029", "\\u2029");

// PHP
$data = str_replace("\xe2\x80\xa8", '\\u2028', $data);
$data = str_replace("\xe2\x80\xa9", '\\u2029', $data);

Or you could just do nothing at all, since PHP escapes non-Unicode characters by default in json_encode():

// Safe
echo json_encode("\xe2\x80\xa9");
--> "\u2029"

// Correct JSON, but invalid Javascript...
// (Well, technically, JSON root must be array or object)
echo json_encode("\xe2\x80\xa9", JSON_UNESCAPED_UNICODE);
--> "
"
hakre
  • 193,403
  • 52
  • 435
  • 836
Dietrich Epp
  • 205,541
  • 37
  • 345
  • 415
  • 1
    Excellent reply, thank-you! I may as well just leave it using json_encode() as that seems to do all the escaping work for me. Good work whoever wrote that function! – zuallauz Jan 06 '13 at 06:25
  • But that assumes UTF-8. Wouldn't that break if you use a different character encoding? – Cole Tobin Aug 20 '16 at 19:30
  • But that assumes UTF-8. Wouldn't that break if you use a different character encoding. – Cole Tobin Aug 20 '16 at 19:33
  • @ColeJohnson: JSON is not permitted to use 8-bit encodings other than UTF-8. – Dietrich Epp Aug 20 '16 at 20:08
  • @DietrichEpp. JSON is absolutely allowed to use any of UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE. There is no need to specify which is used, as it's easy to distinguish, given that the first two characters must be ASCII (the first character will be either `{` or `[`, and the second character will be either `"`, `'`, whitespace, or a digit). – TRiG Jul 14 '17 at 16:38
  • @TRiG: That's correct, but I'm not sure why you are mentioning this? – Dietrich Epp Jul 14 '17 at 16:48
1

It’s worth pointing out that this is no longer necessary.

By default, json_encode() encodes all non-ASCII characters (including U+2028 & U+2029), and also escapes the forward slash, even though that does not need to be escaped by the JSON spec. It does no harm to escape it, and it can be safer in certain contexts. So, by default, these characters are escaped anyway.

The JSON_UNESCAPED_UNICODE constant outputs unescaped Unicode, which can save bytes. However, just as the slash character is escaped because it can be dangerous in some contexts, so too U+2028 & U+2029 are also escaped, because they too are dangerous in some contexts. This was not the case at the time you asked your question: this feature has been added to PHP more recently.

(These extra escapes can be turned off with JSON_UNESCAPED_SLASHES and JSON_UNESCAPED_LINE_TERMINATORS, respectively.)

TRiG
  • 10,148
  • 7
  • 57
  • 107