5

I am high school math teacher who is teaching myself programming. My apologies in advance if I don't phrase some of this correctly.

I am collecting CSV data from the user and trying to move it to a SQLite database via Python.

Everything works fine unless one of the values has a space in it.

For example, here is part of my JavaScript object:

Firstname: "Bruce"
Grade: ""
Lastname: "Wayne Jr"
Nickname: ""

Here is the corresponding piece after applying JSON.stringify:

{"Firstname":"Bruce","Lastname":"Wayne Jr","Nickname":"","Grade":""}

This is then passed to Python via a form. In Python, I use:

data = request.form.get("data")
print(data)
data2 = json.loads(data)
print(data2)

I get a bunch of error messages, ending with: json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 250 (char 249) and the log of the first print gives:

[{"Firstname":"Jason","Lastname":"Bourne","Nickname":"","Grade":"10"},
 {"Firstname":"Steve","Lastname":"McGarret","Nickname":"5-0","Grade":""},
 {"Firstname":"Danny","Lastname":"Williams","Nickname":"Dano","Grade":"12"},
 {"Firstname":"Bruce","Lastname":"Wayne

So it seems to break on the space in "Wayne Jr".

I used what I learned here to build the basics:
https://bl.ocks.org/HarryStevens/0ce529b9b5e4ea17f8db25324423818f

I believe this JavaScript function is parsing the user data:

function changeDataFromField(cb){
   var arr = [];
   $('#enter-data-field').val().replace( /\n/g, "^^^xyz" ).split( "^^^xyz" ).forEach(function(d){
     arr.push(d.replace( /\t/g, "^^^xyz" ).split( "^^^xyz" ))
   });
   cb(csvToJson(arr));
 }

Updates based on comments:
I am using a POST request. No AJAX.

There are actually 2 inputs for the user. A text box where they can paste CSV data and a file upload option. Here is some more of the JavaScript.

// Use the HTML5 File API to read the CSV
  function changeDataFromUpload(evt, cb){
    if (!browserSupportFileUpload()) {
      console.error("The File APIs are not fully supported in this browser!");
    } else {
      var data = null;
      var file = evt.target.files[0];
      var fileName = file.name;
      $("#filename").html(fileName);

      if (file !== "") {
        var reader = new FileReader();

        reader.onload = function(event) {
          var csvData = event.target.result;
          var parsed = Papa.parse(csvData);
          cb(csvToJson(parsed.data));
        };
        reader.onerror = function() {
          console.error("Unable to read " + file.fileName);
        };
      }

      reader.readAsText(file);
      $("#update-data-from-file")[0].value = "";
    }
  }

  // Method that checks that the browser supports the HTML5 File API
  function browserSupportFileUpload() {
    var isCompatible = false;
    if (window.File && window.FileReader && window.FileList && window.Blob) {
      isCompatible = true;
    }
    return isCompatible;
  }

  // Parse the CSV input into JSON
  function csvToJson(data) {
    var cols = ["Firstname","Lastname","Nickname","Grade"];
    var out = [];
    for (var i = 0; i < data.length; i++){
      var obj = {};
      var row = data[i];
      cols.forEach(function(col, index){
        if (row[index]) {
          obj[col] = row[index];
        }
        else {
          obj[col] = "";
        }
      });
      out.push(obj);
    }
    return out;
  }

  //  Produces table for user to check appearance of data and button to complete upload
  function makeTable(data) {
    console.log(data);
    send_data = JSON.stringify(data);
    console.log(send_data);
      var table_data = '<table style="table-layout: fixed; width: 100%" class="table table-striped">';
      table_data += '<th>First name</th><th>Last name</th><th>Nickname</th><th>Grade</th>'
      for(var count = 0; count < data.length; count++) {
        table_data += '<tr>';
            table_data += '<td>'+data[count]['Firstname']+'</td>';
            table_data += '<td>'+data[count]['Lastname']+'</td>';
            table_data += '<td>'+data[count]['Nickname']+'</td>';
            table_data += '<td>'+data[count]['Grade']+'</td>';
        table_data += '</tr>';
        }
      table_data += '</table>';
      table_data += '<p><form action="/uploaded" method="post">';
      table_data += 'Does the data look OK? If so, click to upload.  ';
      table_data += '<button class="btn btn-primary" type="submit">Upload</button><p>';
      table_data += '<input type="hidden" id="data" name="data" value='+send_data+'>';
      table_data += '<input type="hidden" name="class_id" value="{{ class_id }}">';
      table_data += '</form>';
      table_data += 'Otherwise, fix the file and reload.';
      document.getElementById("result_table").innerHTML = table_data;
  }
  </script>
MathGuy297
  • 59
  • 5
  • Are you using an AJAX request to send the data? Spaces aren't valid in a URL. If you are using a GET request, the spaces need to be encoded. It would be better to use a POST request instead. – Tim Roberts Dec 31 '21 at 00:23
  • Very interesting JS. Why would one first replace newlines and then split, instead of splitting on newlines directly. Same for the tabs. This will break your data if someone has ^^^xyz as his nickname. – Thomas Weller Dec 31 '21 at 00:46
  • 1
    The Python error is complaining about the data getting cut off. Oddly enough at the 256th character. I'm about to eat dinner, but that JS appears to just be logging the data. Can you show the actual JavaScript code that you are using. I think the problem might be there. It also looks like the newest version of the papa parse library being used has built in functionality that can replace a lot of what is going on. – hostingutilities.com Dec 31 '21 at 01:00
  • Updated post to address some of the comments. – MathGuy297 Dec 31 '21 at 15:07

2 Answers2

1

The JavaScript can be made a lot simpler. If you do the following instead, do you have any problems with the JSON getting cut off?

document.getElementById('update-data-from-file').addEventListener('change', function(evt){
  var file = evt.target.files[0];
  document.getElementById('filename').innerText = file.name

  if (file === "")
    return;

  Papa.parse(file, {
    header: true,
    skipEmptyLines: true,
    complete: function(results) {
      json = results.data
      console.log(json)
      // Do stuff with the json here
    }
  });

  document.getElementById('update-data-from-file').value = ''
})
<!DOCTYPE html>
<html>
  <head>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/PapaParse/5.0.2/papaparse.min.js"></script>
  </head>
  <body>

    <label>
      Upload CSV
      <input type="file" name="File Upload" id="update-data-from-file" accept=".csv" />
    </label>
    <div id="filename">No file chosen</div>
    
  </body>
</html>
hostingutilities.com
  • 8,894
  • 3
  • 41
  • 51
  • This worked great on the JavaScript side, but I am still getting the same ```json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 103 (char 102)``` error when I try to move it to Python with a POST form. The JSON is still cutting off in Python when it hits a space in one of the values. – MathGuy297 Jan 03 '22 at 17:12
0

The problem was the way I was sending the JSON string -- it wasn't in quotes so anytime there was a space in a value, there was a problem.

To fix it: I got the JSON from the answer above, then before sending the JSON string via a POST request, I enclosed it in quotes.

send_data = JSON.stringify(json);
send_data = "'" + send_data + "'";

I am now able to send values that have spaces in it.

MathGuy297
  • 59
  • 5
  • I'm glad you were able to figure it out. – hostingutilities.com Jan 03 '22 at 22:44
  • 1
    I think the problem is actually at `'';`. The HTML attributes such as `id="data"` have attribute values surrounded by quotes, allowing them to have spaces, but then when you do `value='+send_data+'` the only quotes are the ones ending/starting the JavaScript string, the actual HTML that will result from that won't have surrounding quotes. Your solution adds these surrounding quotes. Does that make sense? – hostingutilities.com Jan 03 '22 at 23:14
  • 1
    The problem isn't with flask. When you have something such as `''` it gets turned into `''`. Chrome will now interpret this as seeing the attribute `a` equal to `this works` and `b` will be equal to `this` and there will be a third attribute called `breaks` that isn't equal to anything. Not what we intended. This is a mistake I sometimes make too. It can be easy to miss seeing that the resulting HTML won't have quotes around an attribute's value. – hostingutilities.com Jan 03 '22 at 23:15
  • @hostingutilities.com Yes, that all makes sense. Thank you very much! I edited my answer to remove the mention of Flask. – MathGuy297 Jan 04 '22 at 23:09