5

Due to some communication errors, I am sometimes receiving JSON strings with some illegal characters: "{messageType\" : \"Test1\", \"from\" : \"F2D0B5C6-9875-46B5-8D4F\"}����1"

These illegal characters are making my JSON parser to break. I am using RapidJSON JSON parser (C/ C++). Can you please tell me if there is a way I can filter these unwanted characters from the string and also verify integrity of the json string.

Deekshith
  • 1,544
  • 2
  • 14
  • 15

5 Answers5

5

It is not a bug in the parser. The parser verifies the trailing characters before null terminator are white spaces. And it returns error code when error happens. But if there is no null terminator, it may cause segmentation fault, similar to strlen().

In the newer versions of RapidJSON, there is a kParseStopWhenDoneFlag. When it is enabled, the parser will stop reading trailing characters after a complete JSON value. E.g.

Document d;
const char* s =
    "{messageType\" : \"Test1\", \"from\" : \"F2D0B5C6-9875-46B5-8D4F\"}����1";
d.Parse<kParseStopWhenDoneFlag>(s);
assert(!d.HasParseError());

By using this flag, the parser will stop after reading }, without reporting error.

It is not yet documented in the guide. Please refer to discussion in https://github.com/miloyip/rapidjson/pull/83

Milo Yip
  • 4,902
  • 2
  • 25
  • 27
2

I think you should consider rolling your own pre-processing function that goes through every character in the JSON string searching for characters that are not part of your legal set and either removes or replaces them with white space. Then pass the newly repaired string forward to RapidJSON.

It's probably better to detect when you've had the comms problems in the first place (and therefore the JSON may be incomplete and or incorrect) and throw away and retry the entire session as opposed to 'patching up' the data as you want to here which solves you short term problem (program crashing) but could easily generate data inconsistencies and other more subtle and difficult to diagnose problems.

Also if you are seeing mostly bad data at the end of a string like this I think you should check carefully that your issue is actually with the comms - the case you give here looks more like a string buffer that was not terminated correctly and has additional junk (uninitialized memory) after the end of the string - perhaps you expected C++ to clear (set to zero) an allocated buffer?

Elemental
  • 7,365
  • 2
  • 28
  • 33
2

File a bug report. A JSON parser should accept whatever input you throw at it and return an appropriate error message. If it crashes, that sounds like a vulnerability that could allow your application to be attacked by hackers. Probably best to find a different parser.

JSON data should never be modified by the receiver to make it work. It should be taken as it is, and if it isn't acceptable data, then it should be refused. If there are "communication errors" that are due to bugs in your code, fix the bug. If they are due to server bugs, complain to whoever wrote the server code. If they are genuine transmission errors, how do you know that you don't have changes that keep the JSON valid, like a payment amount changed from $100 to $900?

gnasher729
  • 51,477
  • 5
  • 75
  • 98
1

You can list the allowed characters in a string and check each character of your json stream is in the allowed_string. Example :

std::string allowed = "abcdefghijklmnopqrstuvwxyz0123456789.,{}[]\"";
std::string json = "{\"bar\":\"foo\",\"blah\":25}";
for(unsigned long int i = 0; i < json.length(); ++i)
  if(allowed.find(json[i]) == std::string::npos)
    throw IllegalJsonChar(json[i]);
Caduchon
  • 4,574
  • 4
  • 26
  • 67
1

If your system is POSIX.1 conformant you can use iconv.

You can either do a system call (Linux/iOS et al):

iconv -f utf-8 -t utf-8 -c file.txt

This will convert from utf-8 to utf-8 skipping any invalid characters.

You can do the same thing using iconv(3), with a bit of programming.

Klas Lindbäck
  • 33,105
  • 5
  • 57
  • 82