Jquery crashes while parsing large csv file

Question

I have this jquery function which I found on a tutorial site that takes in a csv file input and tablifies it. I tried giving in a large csv file(10,000KB) and my browser crashes. I saw there's a parser library called papa to handle this but is there any other approach to keep my browser from crashing while doing this?

Here's the relevant code:-

 $("#upload").bind("click", function () {

        var regex = /^([a-zA-Z0-9\s_\\.\-:])+(.csv|.txt)$/;
        if (regex.test($("#fileUpload").val().toLowerCase())) {
                            var reader = new FileReader();
                reader.onload = function (e) {
                    var table = $("<table id='mytable' class='table table-striped table-bordered'/>");
                    var rows = e.target.result.split("\n");
                    text1=e.target.result;
                    var csvString = $.trim(text1);
                    var csvRows    = csvString.split(/\n/);
                    var csvHeaders = csvRows.shift().split(';');
                    var headerstr=String(csvHeaders);
                    var count = (headerstr.match(/,/g) || []).length+1;
                    for (var i = 0; i < rows.length; i++) {
                        var row = $("<tr />");
                        var cells = rows[i].split(",");
                        for (var j = 0; j < count; j++) {

                            if(cells[j]){
                            var cell = $("<td/>");

                            cell.html(cells[j]);
                            }
                            else{
                            var cell = $("<td class='info'/>");
                            cell.html("empty");}
                            row.append(cell);

                        }
                        table.append(row);

                    }


                    $("#dvCSV").html('');
                    $("#dvCSV").append(table);
                }

    });

How do I implement this functionality without crashing my browser? Thanks in advance.

Answer 1

Two big issues in tackling this problem:

1) The CSV parser. Papa Parse is great. It has support for workers and is a streaming parser - the only way to go with large files.

2) The way you display the data. Simply outputing each row in a table won't work. I crashed my PC twice trying to come up with a working solution. The only way of doing this and the one used by basically any system dealing with large files is to use virtualized lists. I ended up using this one. It's simple and the code is easy to understand.

Here is my JS:

$("#fUpload").bind("change", function(evt) {
    var bigFile = evt.target.files[0];
    var rows = [];
    Papa.parse(bigFile, {
        delimiter: ",",
        newline: "\n",
        header: false,
        dynamicTyping: false,
        worker: false,
        chunk: function(results) {
            rows.concat(rows, results.data);
        },
        complete: function() {
            var list = new VirtualList({
              h: 300,
              itemHeight: 30,
              totalRows: rows.length,
              generatorFn: function(row) {
                  var el = document.createElement("div");
                  el.innerHTML = "<p>ITEM " + row + ' -> ' + rows[row].join(' - ') + "</p>";
                  return el;
              }
            });
            document.body.appendChild(list.container)
        }
    });
});

HTML contains this input: <input type="file" id="fUpload" />

How I configured Papa:

• delimiter and newline: if you allow it to try and detect them it will fail or take longer;

• worker: this will spawn a worker process. It will be slower but will keep the UI responsive (the UI thread won't do any work). You'll probably want to set this to true in production. (This won't work on JSFiddle because of browser cross-domain security protocol!);

• chunk: instead of a callback for each parsed row have one for a larger set of rows. This is faster;

The virtual list config is the default one.

You can run it here.

I tested with a 9.4 MB CSV file containing 1,Foo,100 repeated on each line.

Here's the same but using a table to output the data and added a -1 to VirtualList's totalRows to compensate for the actual length.

Answer 2

I recommend you to use console.log('...') to identify where it's locking/breaking, maybe in the first variables(rows, csvString) because the size. So, you can know where to attack.

If not in the before loop block, then i think it's too expensive use jquery in this context, so try a direct DOM approach(inside the loop,at least):

$("#upload").bind("click", function () {
    var regex = /^([a-zA-Z0-9\s_\\.\-:])+(.csv|.txt)$/;
    if (regex.test($("#fileUpload").val().toLowerCase())) {
                        var reader = new FileReader();
            reader.onload = function (e) {
                var table = $("<table id='mytable' class='table table-striped table-bordered'/>");
                var rows = e.target.result.split("\n");
                text1=e.target.result;
                var csvString = $.trim(text1);
                var csvRows    = csvString.split(/\n/);
                var csvHeaders = csvRows.shift().split(';');
                var headerstr=String(csvHeaders);
                var count = (headerstr.match(/,/g) || []).length+1;
                for (var i = 0; i < rows.length; i++) {
                    var row = document.createElement("tr");
                    var cells = rows[i].split(",");
                    for (var j = 0; j < count; j++) {
                        var cell = document.createElement("td");
                        if(cells[j]){

                        cell.appendChild(document.createTextNode(cells[j]));
                        }
                        else{
                            cell.setAttribute('class','info');
                            cell.appendChild(document.createTextNode("empty"));
                        }
                        row.appendChild(cell);

                    }
                    table.append(row);

                }


                $("#dvCSV").html('');
                $("#dvCSV").append(table);
            }

});

I don't tested this code.