Passing a NaN from Python through Request to Javacscript

Question

I have a pandas array with np.NaN's in it that I convert to a dictionary for JSON

json_data = {"freq" : list(df["Data"])}

I get ready to load to a web page via flask jsonify

@dashboard.route('/command', methods = ['GET', 'POST'])
def command():
[...]
return jsonify(json_data)

and on the javascript side I get ready to read it.

$.ajax({
            url: '/dashboard/command',
            type: 'POST', 
            data: data,
            contentType: 'application/json; charset=utf-8',
            dataType: 'json', 
            success: function(msg) {
              if (!msg.error) {
                updatePlot(msg);
              }
              else {
                alert(msg.error);
              }
            },
            error: function (XMLHttpRequest, textStatus, errorThrown) {
              alert("There has been an error retrieving your data: \n\n" + errorThrown);
            },
            complete: function() {
              $("#update-plot").prop('disabled', false);  // Re-enable button
              $("#update-plot-loading").addClass('invisible');  // Hide loading animation
            }

And that is where it dies if I load that web page. It says that it has a bad token "N" in the json. If I replace the NaN's before trying to send, e.g.

df["Data"].replace(np.NaN, -999.999)

and all is fine and the javascript can continue. I'm aware that the NaN's should be "null" on the javascript side, so what I do at the moment (ugh) is then convert the -999.999's to "null"... But gosh, shouldnt there be a way to send missing data directly?

Any suggestions? I have tried many combinations of NaN, "NaN", None, etc. But maybe I missed one. I checked the output of jsonify on the python said and it seems to be a valid response to send over.

Thanks, T.

`NaN` cannot be encoded as JSON. It's not part of the syntax. — Pointy, Oct 14 '21 at 19:34
If you think about it, `NaN` really isn't "information"; it's not even a specific value. JSON was intended as a data serialization system, so it's arguable that `NaN` isn't really useful data. (The contrary is also arguable, I admit. But that's the way it is.) — Pointy, Oct 14 '21 at 19:51
So it seems like JSON is perfectly happy with null. So maybe if I string replace all NaN's to null it will go through? — Tunneller, Oct 14 '21 at 20:06
Yea, it's up to how your application works. JSON is very limited; for example Date values, which are clearly pretty important in a lot of software, have no standard representation. — Pointy, Oct 14 '21 at 20:08
https://stackoverflow.com/questions/6601812/sending-nan-in-json — Rani Sharim, Oct 14 '21 at 20:54

score 1 · Answer 1 · answered Nov 10 '21 at 15:42

1

Aha. Progress, sort of. At least on the Flask side, maybe not Ajax. Consider a Pandas with some np.Nan in it... Convert to a dict or whatever and then call render_template

s = render_template('dashboard/dashboard_horizontal.html', 
        average_data = average_data)

This is where the damage gets done.... but it is fixable with

s = s.replace("nan", "NaN")      #this is dumb but works!!

A related problem ocurse with false in json. When you read it into Python, the parser conveniently turns it into False. Except now you created problem on the Javascript side, but it is also fixable the same way.

s = s.replace("False", "false")      #this is dumb too

I dont know if there are some keywords that could be passed to render_template to do this automatically?

answered Nov 10 '21 at 15:42

Tunneller

381
2
13

Well, this only works if the **only** occurrences of `nan`/`False` etc are the values... might work in your case but wont work in general – GACy20 Nov 10 '21 at 15:47
Agreed. Although the worst case scenario is words like "inanimate" would mysteriously turn into iNaNimate. But at least it wouldnt crash the browser. – Tunneller Nov 10 '21 at 18:05
1

Ha!! GACy20, I thought you might appreciate this.... Just had a customer complain that the browser not working, turns out I had created the word MataiNaNce in a table that was then subsequently not matching the word "maintainance".. Pretty much exactly as you anticipated... – Tunneller Dec 06 '21 at 18:53

score 0 · Answer 2 · answered Nov 10 '21 at 18:10

Ok, an I have an Ajax solution for NaN too, which is the JSON5 idea explained here : How to parse JSON string containing "NaN" in Node.js . Specifically, dont use dataType json, go with 'text' and do the parsing by hand.

          $.ajax({
            url: '/dashboard/command',
            type: 'POST',   
            data: data,
            contentType: 'application/json; charset=utf-8',
            dataType: 'text',            // it was 'json'
            //async: false,
            success: function(msg) {
              if (!msg.error) {
                var data = json5.parse(msg);  // json5 so much more friendly
                updatePlot(data);
              }
              else {
                alert(msg.error);
              }
            },
            error: function (XMLHttpRequest, textStatus, errorThrown) {
              alert("Error:\n\n" + errorThrown);
            },
            complete: function() {
              $("#update-plot").prop('disabled', false);  // Re-enable button
              $("#update-plot-loading").addClass('invisible');   
            }
          });
        }

Passing a NaN from Python through Request to Javacscript

2 Answers2