Is there a better way to 'doctor' JSON code?

Question

I download some webpage with JSON code embedded into javascript. I need to decode it but it is incorrect JSON and includes single and double quotes which cause error at decode subroutine.

NOTE: JSON extracted as a block into string variable, DATA block represents some form of incorrect JSON code (mostly the problem is in a part which represents input of website visitor clients), JSON has quite deep recursion structure.

So far I could not find better solution than attached bellow code which is still incorrect.

Is there a better way to doctor received JSON code? [May be with (??{ code}) in regex]

use strict;
use warnings;
use diagnostics;

while( <DATA> ) {
    chomp;
    print "IN:  $_\n";
    s/"/'/g;
    print "OUT: $_\n" if s/'(.*?)'\s*:\s*'(.*?)'(,|\s*\})/"$1": "$2"$3/g;
}

__DATA__
{ "d1": "some data here", "d2":"some "data" here", "d3": "some "data" here "year"", "d4": { "x1": "some "data" here" } }
{ "d2": "some data here", "d2":"some "data" here", "d3": "some "data" here "year"" }
{ 'd3': 'some data here', "d2":"some "data" here", "d3": "some "data" here "year"" }
{ "d4": 'some data here', "d2":"some "data" here", "d3": "some "data" here "year"", "d4": { "x1": "some "data" here" } }
{ 'd5': "some data here", "d2":"some "data" here", "d3": "some "data" here "year"" }

output

IN:  { "d1": "some data here", "d2":"some "data" here", "d3": "some "data" here "year"", "d4": { "x1": "some "data" here" } }
OUT: { "d1": "some data here", "d2": "some 'data' here", "d3": "some 'data' here 'year'", "d4': { 'x1": "some 'data' here" } }
IN:  { "d2": "some data here", "d2":"some "data" here", "d3": "some "data" here "year"" }
OUT: { "d2": "some data here", "d2": "some 'data' here", "d3": "some 'data' here 'year'" }
IN:  { 'd3': 'some data here', "d2":"some "data" here", "d3": "some "data" here "year"" }
OUT: { "d3": "some data here", "d2": "some 'data' here", "d3": "some 'data' here 'year'" }
IN:  { "d4": 'some data here', "d2":"some "data" here", "d3": "some "data" here "year"", "d4": { "x1": "some "data" here" } }
OUT: { "d4": "some data here", "d2": "some 'data' here", "d3": "some 'data' here 'year'", "d4': { 'x1": "some 'data' here" } }
IN:  { 'd5': "some data here", "d2":"some "data" here", "d3": "some "data" here "year"" }
OUT: { "d5": "some data here", "d2": "some 'data' here", "d3": "some 'data' here 'year'" }

Answer 1

I'm not going to write a parser for your broken JSON. Teaching how to write a parser is beyond the scope of this site. Besides, you can easily base yours off an existing JSON parser (such as JSON::PP).

What I can do is help you address the only hard part: determining whether a quote ends the literal or needs to be escaped. For example, determining that the second and third quote in "some "data" here" don't end the literal, while the fourth one does.

It turns out it's quite easy to make a reliable guess: Just look ahead! If the quote is followed by optional whitespace whichever of :, ,, } and ] would be valid if the literal ended, the quote probably legitimately ends the literal. Otherwise, it's part of the literal and needs escaping.

Answer 2

It looks like you want to make the following corrections:

1) change any json field names from single-quoted to double-quoted.

2) change any json string field values from single-quoted to double-quoted

3) change any nested double-quotes in json string fields to single-quotes inside the double-quoted string

4) ensure there is at least one space after the colon after a json field name.

Can you double-check the ends of examples 1 and 4 in the output section? Surely you don't want to change the json field name to have a colon and a single-quote and a curly-brace in it. Also, if you did mean to do that, then the curly braces are now unbalanced on those lines.

All that said ... regex may not be the right tool for the job. You may need a context-sensitive parser for this (to keep track of all the levels of nesting).

If you have any control over the page that you are downloading - make them fix it. Otherwise ... you pretty much have to be able to handle anything, so you need the context-sensitive parser - and be prepared to throw away some of the input if it gets too messy. ... instead of crashing or going into an infinite loop.