C++REST SDK, parse JSON that contains (0xEF 0xBF 0xBF) character

Question

I have a REST API that returns a string with files metadata. Among those files I have one that has "" (0xEF0xBF0xBF, link) character in name. When I try to parse such a response I get exception that says:

e = {_message="* Line 1, Column 13 Syntax error: Malformed string literal" }

This is the string I'm trying to parse:

{"files":[["Thunderbolt Gigabit Ethernet Adapter.txt",["bc288518-c426-4dbd-9600-a213a35d1c04",1447772221866,9,"Bartosz","Siewior"]],["System.Windows.Interactivity.dll",["56ce759b-019b-4723-8fca-7af877908971",1440507238241,55904,"Bartosz","Siewior"]]],"folderPermission":[["MODIFY"]],"directories":[]}

The code I'm using to receive and parse the string:

// ...
client.request(request).then([&](web::http::http_response response) {
        status_code = response.status_code();
        //response.extract_json(true).then([&](pplx::task<web::json::value> previousTask) {
        response.extract_string().then([&](pplx::task<utility::string_t> previousTask) {
            try {

                utility::string_t str_response = previousTask.get();
                web::json::value root;
                root = web::json::value::parse(str_response);
            }
            catch (const web::http::http_exception& e) {
                std::wstringstream ss;
                ss << e.what();
                str_response = ss.str();
            }
            catch (const web::json::json_exception& e) {
                std::wstringstream ss;
                ss << e.what();
                str_response = ss.str();
            }

            TaskExecutionData data = { task.id, status_code, str_response.c_str() };
            callback(data);
        }).wait();
    }).wait();

VS2013 JSON Visualizer can parse and show the result properly: VS_json_visualizer

I've tried C++REST version 2.5.0 and 2.6.0, neither of them is able to parse that string... Do you have any ides?

score 0 · Accepted Answer · answered Dec 12 '15 at 09:14

Answer from Casablanca's team:

The issue is that on windows we parse using UTF16 and the particular character you're having issues with maps to 0xFFFF. This is also the character returned by std::char_traits::eof(), so we detect the code point as an "end of stream" signal and terminate the parse appropriately.

The UTF8 parser should not have these issues, so as a workaround you can probably do something similar to how the current web::json::value::parse() function works in json_parsing.cpp:1245, except using char as the template parameter instead of utility::char_t.

C++REST SDK, parse JSON that contains (0xEF 0xBF 0xBF) character

1 Answers1